{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IZ6SNYq_tVVC"
   },
   "source": [
    "# Classify text with BERT\n",
    "\n",
    "### Learning Objectives\n",
    "1. Learn how to load a pre-trained BERT model from TensorFlow Hub\n",
    "2. Learn how to build your own model by combining with a classifier\n",
    "3. Learn how to train a your BERT model by fine-tuning\n",
    "4. Learn how to save your trained model and use it\n",
    "5. Learn how to evaluate a text classification model\n",
    "\n",
    "This lab will show you how to fine-tune BERT to perform sentiment analysis on a dataset of plain-text IMDB movie reviews.\n",
    "In addition to training a model, you will learn how to preprocess text into an appropriate format.\n",
    "\n",
    "# Before you start\n",
    "\n",
    "Please ensure you have a GPU (1 x NVIDIA Tesla K80 should be enough) attached to your Notebook instance to ensure that the training doesn't take too long. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2PHBpLPuQdmK"
   },
   "source": [
    "## About BERT\n",
    "\n",
    "[BERT](https://arxiv.org/abs/1810.04805) and other Transformer encoder architectures have been wildly successful on a variety of tasks in NLP (natural language processing). They compute vector-space representations of natural language that are suitable for use in deep learning models. The BERT family of models uses the Transformer encoder architecture to process each token of input text in the full context of all tokens before and after, hence the name: Bidirectional Encoder Representations from Transformers. \n",
    "\n",
    "BERT models are usually pre-trained on a large corpus of text, then fine-tuned for specific tasks.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SCjmX4zTCkRK"
   },
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:21:56.131967Z",
     "iopub.status.busy": "2021-03-04T02:21:56.131316Z",
     "iopub.status.idle": "2021-03-04T02:21:58.234229Z",
     "shell.execute_reply": "2021-03-04T02:21:58.233522Z"
    },
    "id": "q-YbjCkzw0yU"
   },
   "outputs": [],
   "source": [
    "# A dependency of the preprocessing for BERT inputs\n",
    "!pip install -q --user tensorflow-text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5w_XlxN1IsRJ"
   },
   "source": [
    "You will use the AdamW optimizer from [tensorflow/models](https://github.com/tensorflow/models)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:21:58.241175Z",
     "iopub.status.busy": "2021-03-04T02:21:58.239003Z",
     "iopub.status.idle": "2021-03-04T02:22:19.934874Z",
     "shell.execute_reply": "2021-03-04T02:22:19.934151Z"
    },
    "id": "b-P1ZOA0FkVJ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m  WARNING: The scripts cygdb, cython and cythonize are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "\u001b[33m  WARNING: The script cpuinfo is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "\u001b[33m  WARNING: The script kaggle is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install -q --user tf-models-official"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:22:19.940757Z",
     "iopub.status.busy": "2021-03-04T02:22:19.939965Z",
     "iopub.status.idle": "2021-03-04T02:22:28.247771Z",
     "shell.execute_reply": "2021-03-04T02:22:28.247146Z"
    },
    "id": "_XgTpm9ZxoN9"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import tensorflow_text as text\n",
    "from official.nlp import optimization  # to create AdamW optmizer\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "tf.get_logger().setLevel('ERROR')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check if you have a GPU attached. Run the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num GPUs Available:  1\n"
     ]
    }
   ],
   "source": [
    "print(\"Num GPUs Available: \", len(tf.config.list_physical_devices('GPU')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q6MugfEgDRpY"
   },
   "source": [
    "## Sentiment Analysis\n",
    "\n",
    "This notebook trains a sentiment analysis model to classify movie reviews as *positive* or *negative*, based on the text of the review.\n",
    "\n",
    "You'll use the [Large Movie Review Dataset](https://ai.stanford.edu/~amaas/data/sentiment/) that contains the text of 50,000 movie reviews from the [Internet Movie Database](https://www.imdb.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vnvd4mrtPHHV"
   },
   "source": [
    "### Download the IMDB dataset\n",
    "\n",
    "Let's download and extract the dataset, then explore the directory structure.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:22:28.256239Z",
     "iopub.status.busy": "2021-03-04T02:22:28.255469Z",
     "iopub.status.idle": "2021-03-04T02:22:49.986374Z",
     "shell.execute_reply": "2021-03-04T02:22:49.985726Z"
    },
    "id": "pOdqCMoQDRJL"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
      "84131840/84125825 [==============================] - 2s 0us/step\n"
     ]
    }
   ],
   "source": [
    "url = 'https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz'\n",
    "\n",
    "#Set a path to a folder outside the git repo. This is important so data won't get indexed by git on Jupyter lab\n",
    "path = #example: '/home/jupyter/'\n",
    "\n",
    "dataset = tf.keras.utils.get_file('aclImdb_v1.tar.gz', url,\n",
    "                                  untar=True, cache_dir=path,\n",
    "                                  cache_subdir='')\n",
    "\n",
    "dataset_dir = os.path.join(os.path.dirname(dataset), 'aclImdb')\n",
    "\n",
    "train_dir = os.path.join(dataset_dir, 'train')\n",
    "\n",
    "# remove unused folders to make it easier to load the data\n",
    "remove_dir = os.path.join(train_dir, 'unsup')\n",
    "shutil.rmtree(remove_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lN9lWCYfPo7b"
   },
   "source": [
    "Next, you will use the `text_dataset_from_directory` utility to create a labeled `tf.data.Dataset`.\n",
    "\n",
    "The IMDB dataset has already been divided into train and test, but it lacks a validation set. Let's create a validation set using an 80:20 split of the training data by using the `validation_split` argument below.\n",
    "\n",
    "Note:  When using the `validation_split` and `subset` arguments, make sure to either specify a random seed, or to pass `shuffle=False`, so that the validation and training splits have no overlap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:22:49.993879Z",
     "iopub.status.busy": "2021-03-04T02:22:49.993013Z",
     "iopub.status.idle": "2021-03-04T02:23:02.973133Z",
     "shell.execute_reply": "2021-03-04T02:23:02.972555Z"
    },
    "id": "6IwI_2bcIeX8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 25000 files belonging to 2 classes.\n",
      "Using 20000 files for training.\n",
      "Found 25000 files belonging to 2 classes.\n",
      "Using 5000 files for validation.\n",
      "Found 25000 files belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "AUTOTUNE = tf.data.AUTOTUNE\n",
    "batch_size = 32\n",
    "seed = 42\n",
    "\n",
    "raw_train_ds = tf.keras.preprocessing.text_dataset_from_directory(\n",
    "    path+'aclImdb/train',\n",
    "    batch_size=batch_size,\n",
    "    validation_split=0.2,\n",
    "    subset='training',\n",
    "    seed=seed)\n",
    "\n",
    "class_names = raw_train_ds.class_names\n",
    "train_ds = raw_train_ds.cache().prefetch(buffer_size=AUTOTUNE)\n",
    "\n",
    "val_ds = tf.keras.preprocessing.text_dataset_from_directory(\n",
    "    path+'aclImdb/train',\n",
    "    batch_size=batch_size,\n",
    "    validation_split=0.2,\n",
    "    subset='validation',\n",
    "    seed=seed)\n",
    "\n",
    "val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)\n",
    "\n",
    "test_ds = tf.keras.preprocessing.text_dataset_from_directory(\n",
    "    path+'aclImdb/test',\n",
    "    batch_size=batch_size)\n",
    "\n",
    "test_ds = test_ds.cache().prefetch(buffer_size=AUTOTUNE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HGm10A5HRGXp"
   },
   "source": [
    "Let's take a look at a few reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:02.979717Z",
     "iopub.status.busy": "2021-03-04T02:23:02.978763Z",
     "iopub.status.idle": "2021-03-04T02:23:03.014275Z",
     "shell.execute_reply": "2021-03-04T02:23:03.013684Z"
    },
    "id": "JuxDkcvVIoev"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review: b'\"Pandemonium\" is a horror movie spoof that comes off more stupid than funny. Believe me when I tell you, I love comedies. Especially comedy spoofs. \"Airplane\", \"The Naked Gun\" trilogy, \"Blazing Saddles\", \"High Anxiety\", and \"Spaceballs\" are some of my favorite comedies that spoof a particular genre. \"Pandemonium\" is not up there with those films. Most of the scenes in this movie had me sitting there in stunned silence because the movie wasn\\'t all that funny. There are a few laughs in the film, but when you watch a comedy, you expect to laugh a lot more than a few times and that\\'s all this film has going for it. Geez, \"Scream\" had more laughs than this film and that was more of a horror film. How bizarre is that?<br /><br />*1/2 (out of four)'\n",
      "Label : 0 (neg)\n",
      "Review: b\"David Mamet is a very interesting and a very un-equal director. His first movie 'House of Games' was the one I liked best, and it set a series of films with characters whose perspective of life changes as they get into complicated situations, and so does the perspective of the viewer.<br /><br />So is 'Homicide' which from the title tries to set the mind of the viewer to the usual crime drama. The principal characters are two cops, one Jewish and one Irish who deal with a racially charged area. The murder of an old Jewish shop owner who proves to be an ancient veteran of the Israeli Independence war triggers the Jewish identity in the mind and heart of the Jewish detective.<br /><br />This is were the flaws of the film are the more obvious. The process of awakening is theatrical and hard to believe, the group of Jewish militants is operatic, and the way the detective eventually walks to the final violent confrontation is pathetic. The end of the film itself is Mamet-like smart, but disappoints from a human emotional perspective.<br /><br />Joe Mantegna and William Macy give strong performances, but the flaws of the story are too evident to be easily compensated.\"\n",
      "Label : 0 (neg)\n",
      "Review: b'Great documentary about the lives of NY firefighters during the worst terrorist attack of all time.. That reason alone is why this should be a must see collectors item.. What shocked me was not only the attacks, but the\"High Fat Diet\" and physical appearance of some of these firefighters. I think a lot of Doctors would agree with me that,in the physical shape they were in, some of these firefighters would NOT of made it to the 79th floor carrying over 60 lbs of gear. Having said that i now have a greater respect for firefighters and i realize becoming a firefighter is a life altering job. The French have a history of making great documentary\\'s and that is what this is, a Great Documentary.....'\n",
      "Label : 1 (pos)\n"
     ]
    }
   ],
   "source": [
    "for text_batch, label_batch in train_ds.take(1):\n",
    "  for i in range(3):\n",
    "    print(f'Review: {text_batch.numpy()[i]}')\n",
    "    label = label_batch.numpy()[i]\n",
    "    print(f'Label : {label} ({class_names[label]})')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dX8FtlpGJRE6"
   },
   "source": [
    "## Loading models from TensorFlow Hub\n",
    "\n",
    "For the purpose of this lab, we will be loading a model called Small BERT. Small BERT has the same general architecture as the original BERT but the has fewer and/or smaller Transformer blocks. \n",
    "\n",
    "Some other popular BERT models are BERT Base, ALBERT, BERT Experts, Electra. See the continued learning section at the end of this lab for more info. \n",
    "\n",
    "Aside from the models available below, there are [multiple versions](https://tfhub.dev/google/collections/transformer_encoders_text/1) of the models that are larger and can yeld even better accuracy but they are too big to be fine-tuned on a single GPU. You will be able to do that on the [Solve GLUE tasks using BERT on a TPU colab](https://www.tensorflow.org/tutorials/text/solve_glue_tasks_using_bert_on_tpu).\n",
    "\n",
    "You'll see in the code below that switching the tfhub.dev URL is enough to try any of these models, because all the differences between them are encapsulated in the SavedModels from TF Hub."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choose a BERT model to fine-tune"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:03.027533Z",
     "iopub.status.busy": "2021-03-04T02:23:03.026458Z",
     "iopub.status.idle": "2021-03-04T02:23:03.029361Z",
     "shell.execute_reply": "2021-03-04T02:23:03.029774Z"
    },
    "id": "y8_ctG55-uTX"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BERT model selected           : https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/1\n",
      "Preprocess model auto-selected: https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3\n"
     ]
    }
   ],
   "source": [
    "bert_model_name = 'small_bert/bert_en_uncased_L-4_H-512_A-8'  \n",
    "\"\"\"\n",
    "@param [\"bert_en_uncased_L-12_H-768_A-12\", \n",
    "\"bert_en_cased_L-12_H-768_A-12\", \"bert_multi_cased_L-12_H-768_A-12\", \n",
    "\"small_bert/bert_en_uncased_L-2_H-128_A-2\", \n",
    "\"small_bert/bert_en_uncased_L-2_H-256_A-4\", \n",
    "\"small_bert/bert_en_uncased_L-2_H-512_A-8\", \n",
    "\"small_bert/bert_en_uncased_L-2_H-768_A-12\", \n",
    "\"small_bert/bert_en_uncased_L-4_H-128_A-2\", \n",
    "\"small_bert/bert_en_uncased_L-4_H-256_A-4\", \n",
    "\"small_bert/bert_en_uncased_L-4_H-512_A-8\", \n",
    "\"small_bert/bert_en_uncased_L-4_H-768_A-12\", \n",
    "\"small_bert/bert_en_uncased_L-6_H-128_A-2\", \n",
    "\"small_bert/bert_en_uncased_L-6_H-256_A-4\", \n",
    "\"small_bert/bert_en_uncased_L-6_H-512_A-8\", \n",
    "\"small_bert/bert_en_uncased_L-6_H-768_A-12\", \n",
    "\"small_bert/bert_en_uncased_L-8_H-128_A-2\", \n",
    "\"small_bert/bert_en_uncased_L-8_H-256_A-4\", \n",
    "\"small_bert/bert_en_uncased_L-8_H-512_A-8\", \n",
    "\"small_bert/bert_en_uncased_L-8_H-768_A-12\", \n",
    "\"small_bert/bert_en_uncased_L-10_H-128_A-2\", \n",
    "\"small_bert/bert_en_uncased_L-10_H-256_A-4\", \n",
    "\"small_bert/bert_en_uncased_L-10_H-512_A-8\", \n",
    "\"small_bert/bert_en_uncased_L-10_H-768_A-12\", \n",
    "\"small_bert/bert_en_uncased_L-12_H-128_A-2\", \n",
    "\"small_bert/bert_en_uncased_L-12_H-256_A-4\", \n",
    "\"small_bert/bert_en_uncased_L-12_H-512_A-8\", \n",
    "\"small_bert/bert_en_uncased_L-12_H-768_A-12\", \n",
    "\"albert_en_base\", \"electra_small\", \n",
    "\"electra_base\", \n",
    "\"experts_pubmed\", \n",
    "\"experts_wiki_books\", \n",
    "\"talking-heads_base\"]\n",
    "\"\"\"\n",
    "map_name_to_handle = {\n",
    "    'bert_en_uncased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3',\n",
    "    'bert_en_cased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_cased_L-12_H-768_A-12/3',\n",
    "    'bert_multi_cased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_multi_cased_L-12_H-768_A-12/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-2_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-2_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-2_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-4_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-4_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-4_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-4_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-6_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-6_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-6_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-6_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-8_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-8_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-8_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-8_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-10_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-10_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-10_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-10_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-12_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-12_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-12_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-768_A-12/1',\n",
    "    'albert_en_base':\n",
    "        'https://tfhub.dev/tensorflow/albert_en_base/2',\n",
    "    'electra_small':\n",
    "        'https://tfhub.dev/google/electra_small/2',\n",
    "    'electra_base':\n",
    "        'https://tfhub.dev/google/electra_base/2',\n",
    "    'experts_pubmed':\n",
    "        'https://tfhub.dev/google/experts/bert/pubmed/2',\n",
    "    'experts_wiki_books':\n",
    "        'https://tfhub.dev/google/experts/bert/wiki_books/2',\n",
    "    'talking-heads_base':\n",
    "        'https://tfhub.dev/tensorflow/talkheads_ggelu_bert_en_base/1',\n",
    "}\n",
    "\n",
    "map_model_to_preprocess = {\n",
    "    'bert_en_uncased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'bert_en_cased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_cased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-4_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-4_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-4_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-4_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-6_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-6_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-6_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-6_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-8_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-8_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-8_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-8_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-10_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-10_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-10_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-10_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-12_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-12_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-12_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'bert_multi_cased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_multi_cased_preprocess/3',\n",
    "    'albert_en_base':\n",
    "        'https://tfhub.dev/tensorflow/albert_en_preprocess/3',\n",
    "    'electra_small':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'electra_base':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'experts_pubmed':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'experts_wiki_books':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'talking-heads_base':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "}\n",
    "\n",
    "tfhub_handle_encoder = map_name_to_handle[bert_model_name]\n",
    "tfhub_handle_preprocess = map_model_to_preprocess[bert_model_name]\n",
    "\n",
    "print(f'BERT model selected           : {tfhub_handle_encoder}')\n",
    "print(f'Preprocess model auto-selected: {tfhub_handle_preprocess}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7WrcxxTRDdHi"
   },
   "source": [
    "## The preprocessing model\n",
    "\n",
    "Text inputs need to be transformed to numeric token ids and arranged in several Tensors before being input to BERT. TensorFlow Hub provides a matching preprocessing model for each of the BERT models discussed above, which implements this transformation using TF ops from the TF.text library. It is not necessary to run pure Python code outside your TensorFlow model to preprocess text.\n",
    "\n",
    "The preprocessing model must be the one referenced by the documentation of the BERT model, which you can read at the URL printed above. For BERT models from the drop-down above, the preprocessing model is selected automatically.\n",
    "\n",
    "Note: You will load the preprocessing model into a [hub.KerasLayer](https://www.tensorflow.org/hub/api_docs/python/hub/KerasLayer) to compose your fine-tuned model. This is the preferred API to load a TF2-style SavedModel from TF Hub into a Keras model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 1: Use hub.KerasLaye to initialize the preprocessing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_preprocess_model = hub.KerasLayer(tfhub_handle_preprocess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x4naBiEE_cZX"
   },
   "source": [
    "Let's try the preprocessing model on some text and see the output:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 2: Call the preprocess model function and pass text_test**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:06.221646Z",
     "iopub.status.busy": "2021-03-04T02:23:06.218007Z",
     "iopub.status.idle": "2021-03-04T02:23:06.409091Z",
     "shell.execute_reply": "2021-03-04T02:23:06.408411Z"
    },
    "id": "r9-zCzJpnuwS"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keys       : ['input_word_ids', 'input_type_ids', 'input_mask']\n",
      "Shape      : (1, 128)\n",
      "Word Ids   : [ 101 2023 2003 2107 2019 6429 3185  999  102    0    0    0]\n",
      "Input Mask : [1 1 1 1 1 1 1 1 1 0 0 0]\n",
      "Type Ids   : [0 0 0 0 0 0 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "text_test = ['this is such an amazing movie!']\n",
    "text_preprocessed = bert_preprocess_model(text_test)\n",
    "\n",
    "#This print box will help you inspect the keys in the pre-processed dictionary\n",
    "print(f'Keys       : {list(text_preprocessed.keys())}')\n",
    "\n",
    "# 1. input_word_ids is the ids for the words in the tokenized sentence\n",
    "print(f'Shape      : {text_preprocessed[\"input_word_ids\"].shape}')\n",
    "print(f'Word Ids   : {text_preprocessed[\"input_word_ids\"][0, :12]}')\n",
    "\n",
    "#2. input_mask is the tokens which we are masking (masked language model)\n",
    "print(f'Input Mask : {text_preprocessed[\"input_mask\"][0, :12]}')\n",
    "\n",
    "#3. input_type_ids is the sentence id of the input sentence.\n",
    "print(f'Type Ids   : {text_preprocessed[\"input_type_ids\"][0, :12]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EqL7ihkN_862"
   },
   "source": [
    "As you can see, now you have the 3 outputs from the preprocessing that a BERT model would use (`input_words_id`, `input_mask` and `input_type_ids`).\n",
    "\n",
    "Some other important points:\n",
    "- The input is truncated to 128 tokens. \n",
    "- The `input_type_ids` only have one value (0) because this is a single sentence input. For a multiple sentence input, it would have one number for each input.\n",
    "\n",
    "Since this text preprocessor is a TensorFlow model, It can be included in your model directly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DKnLPSEmtp9i"
   },
   "source": [
    "## Using the BERT model\n",
    "\n",
    "Before putting BERT into your own model, let's take a look at its outputs. You will load it from TF Hub and see the returned values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:06.415829Z",
     "iopub.status.busy": "2021-03-04T02:23:06.414780Z",
     "iopub.status.idle": "2021-03-04T02:23:15.391736Z",
     "shell.execute_reply": "2021-03-04T02:23:15.391122Z"
    },
    "id": "tXxYpK8ixL34"
   },
   "outputs": [],
   "source": [
    "bert_model = hub.KerasLayer(tfhub_handle_encoder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:15.398942Z",
     "iopub.status.busy": "2021-03-04T02:23:15.398146Z",
     "iopub.status.idle": "2021-03-04T02:23:16.055854Z",
     "shell.execute_reply": "2021-03-04T02:23:16.055199Z"
    },
    "id": "_OoF9mebuSZc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded BERT: https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/1\n",
      "Pooled Outputs Shape:(1, 512)\n",
      "Pooled Outputs Values:[ 0.7626287   0.9928097  -0.18611862  0.3667387   0.15233767  0.6550448\n",
      "  0.9681154  -0.948627    0.00216148 -0.98777324  0.06842747 -0.97630584]\n",
      "Sequence Outputs Shape:(1, 128, 512)\n",
      "Sequence Outputs Values:[[-0.28946355  0.34321284  0.3323149  ...  0.21300822  0.71020937\n",
      "  -0.05771076]\n",
      " [-0.28741962  0.31980997 -0.23018558 ...  0.5845511  -0.21329674\n",
      "   0.72692055]\n",
      " [-0.66156894  0.68876857 -0.87433094 ...  0.10877182 -0.26173127\n",
      "   0.47855434]\n",
      " ...\n",
      " [-0.22561201 -0.28925633 -0.07064426 ...  0.4756598   0.83277243\n",
      "   0.40025336]\n",
      " [-0.29824105 -0.27473113 -0.05450625 ...  0.48849684  1.0955372\n",
      "   0.18163362]\n",
      " [-0.4437816   0.00930727  0.07223672 ...  0.17290097  1.1833246\n",
      "   0.07898031]]\n"
     ]
    }
   ],
   "source": [
    "bert_results = bert_model(text_preprocessed)\n",
    "\n",
    "print(f'Loaded BERT: {tfhub_handle_encoder}')\n",
    "print(f'Pooled Outputs Shape:{bert_results[\"pooled_output\"].shape}')\n",
    "print(f'Pooled Outputs Values:{bert_results[\"pooled_output\"][0, :12]}')\n",
    "print(f'Sequence Outputs Shape:{bert_results[\"sequence_output\"].shape}')\n",
    "print(f'Sequence Outputs Values:{bert_results[\"sequence_output\"][0, :12]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sm61jDrezAll"
   },
   "source": [
    "The BERT models return a map with 3 important keys: `pooled_output`, `sequence_output`, `encoder_outputs`:\n",
    "\n",
    "- `pooled_output` to represent each input sequence as a whole. The shape is `[batch_size, H]`. You can think of this as an embedding for the entire movie review.\n",
    "- `sequence_output` represents each input token in the context. The shape is `[batch_size, seq_length, H]`. You can think of this as a contextual embedding for every token in the movie review.\n",
    "- `encoder_outputs` are the intermediate activations of the `L` Transformer blocks. `outputs[\"encoder_outputs\"][i]` is a Tensor of shape `[batch_size, seq_length, 1024]` with the outputs of the i-th Transformer block, for `0 <= i < L`. The last value of the list is equal to `sequence_output`.\n",
    "\n",
    "For the fine-tuning you are going to use the `pooled_output` array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pDNKfAXbDnJH"
   },
   "source": [
    "## Define your model\n",
    "\n",
    "You will create a very simple fine-tuned model, with the preprocessing model, the selected BERT model, one Dense and a Dropout layer.\n",
    "\n",
    "Note: for more information about the base model's input and output you can use just follow the model's url for documentation. Here specifically you don't need to worry about it because the preprocessing model will take care of that for you.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 3: Define your model. It should contain the preprocessing model, the selected BERT model (smallBERT), a dense layer and dropout layer**\n",
    "\n",
    "**HINT** The order of the layers in the model should be:\n",
    "1. Input Layer\n",
    "2. Pre-processing Layer\n",
    "3. Encoder Layer\n",
    "4. From the BERT output map, use pooled_output\n",
    "5. Dropout layer\n",
    "6. Dense layer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:16.063451Z",
     "iopub.status.busy": "2021-03-04T02:23:16.062660Z",
     "iopub.status.idle": "2021-03-04T02:23:16.065220Z",
     "shell.execute_reply": "2021-03-04T02:23:16.064597Z"
    },
    "id": "aksj743St9ga"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[0.66593003]], shape=(1, 1), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "def build_classifier_model():\n",
    "  text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')\n",
    "  preprocessing_layer = hub.KerasLayer(tfhub_handle_preprocess, name='preprocessing')\n",
    "  encoder_inputs = preprocessing_layer(text_input)\n",
    "  encoder = hub.KerasLayer(tfhub_handle_encoder, trainable=True, name='BERT_encoder')\n",
    "  outputs = encoder(encoder_inputs)\n",
    "  net = outputs['pooled_output']\n",
    "  net = tf.keras.layers.Dropout(0.1)(net)\n",
    "  net = tf.keras.layers.Dense(1, activation=None, name='classifier')(net)\n",
    "  return tf.keras.Model(text_input, net)\n",
    "\n",
    "#Let's check that the model runs with the output of the preprocessing model.\n",
    "classifier_model = build_classifier_model()\n",
    "bert_raw_result = classifier_model(tf.constant(text_test))\n",
    "print(tf.sigmoid(bert_raw_result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZTUzNV2JE2G3"
   },
   "source": [
    "The output is meaningless, of course, because the model has not been trained yet.\n",
    "\n",
    "Let's take a look at the model's structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:23.638952Z",
     "iopub.status.busy": "2021-03-04T02:23:23.638165Z",
     "iopub.status.idle": "2021-03-04T02:23:23.873402Z",
     "shell.execute_reply": "2021-03-04T02:23:23.873887Z"
    },
    "id": "0EmzyHZXKIpm"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASgAAAHBCAIAAABVLrEEAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3de1QTZ/4/8M+EhJAECIFiQPCCnla7uzG2aisojYgIHrEIXwWt1221rrRr1drLftuv3z3F029btfaGZbW7ZXuOp4LdlkqlrVbRU26ntFW0rYCXdRW5BZBL5CKX+f0xv52dBgnhIg+D79dfzDNPZj55kncy84RkOJ7nCQCGloJ1AQB3IwQPgAEED4ABBA+AASXrAvpm6dKlrEuAYerQoUOsS+gDTl6zmhzHzZw5MzAwkHUhMIyUlZUVFBTI7Jkss3I5Li0tLT4+nnUhMIykp6cnJCTI65mMczwABhA8AAYQPAAGEDwABhA8AAYQPAAGEDwABhA8AAYQPAAGEDwABhA8AAYQPAAGEDwABhC8IeLu7s5J7Nq1i3VF/zGcaxupELwhYrPZTp8+TUQxMTE8z2/bto11Rf8xnGsbqUZm8Nzd3WfPni2vLQ8Nudc/YozM4AEMcwgeAAMjLXi7du3iOO7mzZu5ubnCVIFS+Z8fdLJarZs2bRo/fryrq6uvr29cXNyZM2eEVbNnzxZnF1auXElE8+bNE1vq6+sdb7l/MjIyxF1cuXIlISHBy8vLx8cnOjr60qVL0nvEcVxgYGBhYWF4eLiHh4dWqw0LC8vNzRX67NixQ+gjHkZ+9dVXQss999zjzMg4o6OjIy0tLSIiws/PT6PRmEymt99+u6uri4jq6+ul0zM7duwQ+ostS5YsETbi4CGQjkZJSUl8fLyPj4+wWFNTM6CBHoZ4WSGitLS0XrvpdLpZs2bZNZaXl48bN85oNB45cqSpqemnn36yWCxubm55eXlChzNnzuh0OrPZbLPZeJ5vbW19+OGHP/744163zPN8WFiYt7d3fn6+g6qkExhSMTExQnteXp7NZjt27JhGo5kxY4a0j9ls1ul0wcHBQp/CwsIpU6a4urqePHnSQW3Tpk3z8fFxpv6eapPKzMwkoldffbWurs5qtb7zzjsKhWLbtm1ih8jISIVCcfHiRemtgoODDxw4IPzd60MgjobFYsnOzr5582ZBQYGLi4vVanVQWFpamvyeyawL6JuBBG/NmjVEJD4JeJ6vqKhQq9XTpk0TW9LT04koLi6uq6trzZo1//3f/+3Mlnmet1gsBoNB+gTqznHwMjMzxRbh/UH6bDObzUR0+vRpseXs2bNEZDabHdQ26MGbM2eOtGXlypUqlaqhoUFY/Prrr4koMTFR7JCTkxMQEHDr1i1h0ZmHQBiNrKwsB5XYkWPwRtqhpgMZGRkKhSI6Olps8fPz++1vf/vDDz+UlZUJLUuXLn3ppZc+/fTT2bNn19bWJiUlObnxkydP1tXVBQcH97u8GTNmiH+PGTOGiMrLy6UddDrd1KlTxUWTyTR69OiioqKKiop+77RPoqOjs7OzpS1ms7m9vf3nn38WFufPn28ymVJTU2tra4WWnTt3/vGPf1SpVMKiMw+B4KGHHrqD92QYuFuC19bW1tDQ0NXVpdfrpWcjP/74IxFduHBB7JmUlPTwww/n5eUtXbpUoRi68dHr9eLfrq6uRCScPom8vLzsbjJq1Cgiqq6uvvPVERE1NDRs377dZDIZDAZh9J577jkiam5uFvts3ry5ubl57969RFRaWnrixIknn3xSWOX8Q0BEOp1uaO4UKyMzeBzH2bWo1WovLy+lUtne3t79fT8sLEzsefLkyYaGBpPJlJiYWFRU1OuWh0xtbS3/65+OFCInxI+IFArFrVu3pB3q6+vtNjKQ+hctWpSUlLR+/frS0tKuri6e5/fs2UNE0qpWrFhhNBrfe++9tra23bt3r1mzxmAwCKucfwjuBiMzeFqtVnwKTpo0ad++fUQUFxfX0dEhzgQKXn/99bFjx3Z0dAiL//znP5944ol//OMfhw8f1mg0MTExVqu11y0PjdbW1sLCQnHx3Llz5eXlZrPZ399faPH3979+/brYobKy8urVq3Yb6Uf9SqWyuLi4s7MzNzfXz89v06ZNvr6+QoBbWlrsOqvV6sTExOrq6t27dx84cOCZZ56RrnXmIbhb3OFzyEFGzk2uREVF6fX6q1ev5uXlKZXKX375hef5qqqqiRMnTpgwISsrq76+vra2NiUlRavVihtsamqaMmXK559/LiyePHlSpVI98sgj4txAT1vmB2NWs6WlRWx54YUX6NdTKWazWa/Xh4eHO5jVfPrpp4no3XffbWpqunjxYnx8fEBAgN3kSk/1O5hccXFxOX/+PM/zc+fOJaI33njDarU2NzefOHFi7NixRHTs2DFpf6vVqtFoOI7rvrVeH4Lbjkav5Di5IrdynQtecXFxaGioTqcbM2ZMcnKy2F5bW7t169YJEyaoVCpfX9/58+eLT5qnnnpKfDE6d+6c3RtdUlKS4y2HhoY6ntW0O2nZuXMnz/P5+fnSxpdeeon/9cHkwoULhZubzeaAgIBffvklMjLSw8NDo9FYLJacnBzpLurr69etW+fv76/RaGbPnl1YWDht2jRhOy+88IKD+ns9oRKCZ7VaN2zYMGbMGJVKZTQa165d++KLLwodpNOSPM+vX7+eiE6dOtV9HBw8BHaj4XyWELw7zsngjTxC8FhX4ay//e1vdlG8o+QYvJF5jgdspaSkbN26lXUVwxqCB4Pjgw8+iI2NtdlsKSkpN27cwBWdHEPwhjvhfyyLioquX7/OcdzLL7/MuqIeZWRkGAyG999//+DBgwP/R9aRDdfHA9nD9fEAwCkIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAPy+3bCzJkzAwMDWRcCw0hZWVlBQYHMnsnyKnfp0qWsSxguvv/+eyKaPn0660KGi0OHDrEuoQ9kFjwQCV9KFH5zHmQH53gADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAM4IqwspGamvrWW291dnYKi1arlYh8fX2FRRcXl82bN69du5ZVedAnCJ5slJSUTJ482UGH8+fPO+4AwwcONWVj0qRJJpOJ47juqziOM5lMSJ2MIHhysnr1ahcXl+7tSqVyzZo1Q18P9BsONeWkvLw8MDCw+0PGcdzVq1cDAwOZVAX9gHc8ORk9enRISIhC8atHTaFQhISEIHXyguDJzKpVq+xO8ziOW716Nat6oH9wqCkzdXV1RqOxo6NDbHFxcamqqvLx8WFYFfQV3vFkxtvbOyIiQqlUCosuLi4RERFInewgePKzcuXKrq4u4W+e51etWsW2HugHHGrKz82bN++5557W1lYiUqvVNTU17u7urIuCvsE7nvzodLpHH31UpVIplcrFixcjdXKE4MnSihUrOjo6Ojs7H3vsMda1QH8oe1qRnp4+lHVAn3R2drq5ufE8b7PZ8EgNZ/Hx8bdt7/Ec77b/EwgAfdJTvnp8xyOitLS0nvIKzGVnZ3McN2fOHNaFwO2lp6cnJCT0tNZR8GA4s1gsrEuA/kPw5MruPzZBXvDgATCA4AEwgOABMIDgATCA4AEwgOABMIDgATCA4AEwgOABMIDgATCA4AEwgODdpdzd3TmJXbt2Ce2TJ08WG2fPnj0MKxwZEDxmbDbbvffeGx0dzWrvp0+fJqKYmBie57dt2ya0Z2dnT506de3ate3t7Tk5OUxqc1zhyIDgMcPzfFdXl/h7YcNBcXFxSEhIdHT0hx9+KP6CINwJGFxmPDw8Ll26xLqK/8jNzY2Li0tKSnryySdZ1zLyIXhARPTpp58++eSTqamprA597zb9P9TctWuXcNYbGBhYWFgYHh7u4eGh1WrDwsJyc3OFPhkZGeLJcUlJSXx8vI+Pj7BYU1NDRFarddOmTePHj3d1dfX19Y2Liztz5swgbr+2tnbr1q0TJ050dXU1GAwLFizIzs6W3guxg1qtDgwMnDdvXmpqaktLi9jBQYVE1NbWtn379smTJ2u1Wm9v70WLFh0+fFi8aKuDtdLKhV/IlLZcuXIlISHBy8vLx8cnOjra7o2xuLh48eLFer1eq9U+9NBDX3zxxbx584Qbrlu3rh8P5XvvvZeYmJiVlXXb1DkYAcfj39HRkZaWFhER4efnp9FoTCbT22+/LT20djx6TnKwl/r6eun0zI4dO4T+YsuSJUsGeB/7Ptj/xveAiNLS0npaKzKbzTqdLjg4OC8vz2azFRYWTpkyxdXV9eTJk2KfmJgYIrJYLNnZ2Tdv3iwoKHBxcbFareXl5ePGjTMajUeOHGlqavrpp58sFoubm1teXt6gbL+ioiIoKMhoNGZmZjY0NJSUlMTFxXEct3//fuGGQgc/P7/MzMzGxsbKysqkpCQi2rNnj9Ch1wrXrVun1+uPHj3a3NxcWVkpTABkZ2c7s1asvKWlxa4lJiZGuL/Hjh3TaDQzZswQO1y4cMHLyysgIODo0aNCSfPmzfP19VWr1dLHJSwszNvbOz8/38FjJ0xdCD/L+eyzz962jzOPUU/jn5mZSUSvvvpqXV2d1Wp95513FArFtm3bxBv2Oj7SyZWe9LqXyMhIhUJx8eJF6a2Cg4MPHDgw8PvooLC0tDRH+epxhdPBI6LTp0+LLWfPniUis9lsV3RWVpbdbYVrKYr3n+f5iooKtVo9bdq0Qdm+cEHwjz/+WGxpbW0dPXq0RqOprKwUO9jdzaioKDF4vVYYFBQUEhIivfl9990nPnUcr+V7Dl5mZqbYIrwqi4/x0qVLieiTTz4RO1RXV2u1WrvgWSwWg8Egfep0JzytJ02a5OnpSUQ7d+7s3seZx6in8c/MzJwzZ460ZeXKlSqVqqGhQVjsdXycDJ7jvXz99ddElJiYKHbIyckJCAi4devWwO+jA3c8eDqdzq5x9OjRRFReXi4sCkXX1NTYddPr9QqFQhwgwYMPPkhE165dG5TtE1FjY6O0UbjSwN///veeOvSpwo0bNxLR+vXr8/PzOzo67G7ueC3fc/CE1wXBli1biKioqEhY9PDwIKKmpia7kuyC5wzxaZ2Xlydsdvfu3X0dAb7n8e9u586dRCS+HPQ6Ps4Er9e98DxvMpm0Wq1YYUxMzGuvvXaH7qPIcfAG4eMELy8vu5ZRo0YRUXV1tbRRp9NJF9va2hoaGrq6uvR6vfRA/McffySiCxcuDMr23dzchKeUyGg0ElFlZWVPHfpUYXJy8kcffXT58uXw8HBPT8+oqKjPPvtM3ILjtQ4IrwgCV1dXIhJOWtra2pqamtzc3Ox+tt1gMDiz2Z4EBwd/+eWX7u7uzz777FtvvdWnERDZjT8RNTQ0bN++3WQyGQwG4YbPPfccETU3Nwsd+j0+fdoLEW3evLm5uXnv3r1EVFpaeuLECXHmdoD3sd8GIXi1tbX8r3+1U4iEEI+eqNVqLy8vpVLZ3t7e/fUgLCxs4NvX6/Wtra1NTU3S9qqqKiLy8/PrqUOfKuQ4btWqVd988019fX1GRgbP83FxcW+++aawBcdr+0GtVnt4eLS2ttpstu4DMhCzZs3KysrS6XRbtmx59913xd05+Rjd1qJFi5KSktavX19aWtrV1cXz/J49e0jyG6+DMj697oWIVqxYYTQa33vvvba2tt27d69Zs0Z8qRrgfey3QQhea2trYWGhuHju3Lny8nKz2ezv7+/4hnFxcR0dHeIUpeD1118fO3as9MKL/d5+bGwsER05ckRsaWtrO378uEajiYyMFDtkZWVJb/XAAw8IR3fOVOjl5VVcXExEKpUqIiJCmAET9+h4bf8sWLCAiL766iuxpbKysrS0dCDbFISGhh45ckSr1W7atCk5OVlodPIx6q6zszM3N9fPz2/Tpk2+vr4cxxGRdLqYBjY+SqWyuLjYmb0QkVqtTkxMrK6u3r1794EDB5555hnp2n7fxwHp6RiUnD7H0+v14eHhvc46Ss9kBFVVVRMnTpwwYUJWVlZ9fX1tbW1KSopWq5XudyDbl85qNjY2irOa+/btk3bw9/f/4osvGhsbr127tnHjRqPR+K9//cvJCvV6vcViKSoqam1traqq+vOf/0xEO3bscGbtbSvv3vLCCy+QZHrp4sWL3t7e4qzmuXPnoqKixo0b1+9ZTbszqBMnTmg0GiJKTk528jHqafznzp1LRG+88YbVam1ubj5x4sTYsWOJ6NixY06Oj4NzPBcXl/PnzzuzF4HVatVoNBzHdd/aQO6jA3d8ciUgIOCXX36JjIz08PDQaDQWiyUnJ0dYm5+f7zjnwsdoEyZMUKlUvr6+8+fPtxuvAW6/pqZm8+bNQUFBKpVKr9dHRkYeP368pw7+/v7Lli0rLS11vsIzZ85s2LDh/vvvFz6Jmjlz5v79+4UDHsdr7U5mVqxYYXdfXnrpJf7XB9gLFy4UNltSUrJ48WJPT0+tVhsSEnLq1Kk5c+ZotVpp2aGhoY5nNe1OV6RTmt98842QPSJKSkpyMAKOx99qtW7YsGHMmDEqlcpoNK5du/bFF18UugkTho5Hr9cTKiF4ve5FtH79eiI6depU99Ho9310YCiC52Qp/XCntz8yTJo0aezYsayrGO7+9re/2UXxjrrjs5owlCorK729vdvb28WWK1euXLp0STjiAgdSUlK2bt3Kuor/D8GTnxs3bmzYsOHatWvNzc3fffddQkKCp6fn//zP/7Cuazj64IMPYmNjbTZbSkrKjRs3hs/Vrwb6v5pFRUXXr1/nOO7ll18exLKGYPsy5efnJ8y/P/LIIwaD4dFHH7333nu/++67CRMmsC5tmMrIyDAYDO+///7BgweHz3edHF2YEtfHA+g34fp4PeULh5oADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADDj6lkT3L70DgJMcx8fR14LuTD0Ad5Ee89XTChjmhK9Kpqensy4E+gPneAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAw4ugY6DCunTp0qKCgQF4uLi4no9ddfF1tmzpxpsVgYVAZ9h0sxy8axY8fmz5+vUqkUCvvjlK6urvb29qNHj0ZERDCpDfoKwZONzs5Oo9FYW1t727UGg6G6ulqpxCGMPOAcTzZcXFxWrFjh6urafZWrq+uqVauQOhlB8ORk+fLlt27d6t5+69at5cuXD3090G841JSZcePGXb161a4xMDDw6tWrHMcxKQn6Ae94MrNy5UqVSiVtcXV1XbNmDVInL3jHk5nz58//5je/sWs8d+7c7373Oyb1QP8gePLzm9/85vz58+Li5MmTpYsgCzjUlJ/Vq1eLR5sqlWrNmjVs64F+wDue/Fy9enX8+PHCA8dx3OXLl8ePH8+6KOgbvOPJz9ixY6dPn65QKDiOmzFjBlInRwieLK1evVqhULi4uKxatYp1LdAfONSUJavV6u/vT0TXr183Go2sy4G+4yXS0tJYlwMwMqWlpUmzdpv/7kP8ZOHUqVMcxz3yyCOsC4HeJSQk2LXcJnjx8fFDUgwMSFRUFBF5enqyLgR651TwQBYQOVnDrCYAAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgAD/Qmeu7s7141WqzWbzW+++WZnZ6fjnlLff/+9g55ubm5TpkxJTk4Wv607depUxxvkOG7Hjh0DHxcmDh48KN7xIdup3cjv2rVLaJ88ebLYOHv27CGrx/kKZa0/wbPZbKdPnyaimJgY4Vt9jY2NX331FRE9++yzzz33nIOeUnq93kHPtra2goICT0/Pp59++oUXXhB7Hjp0SNzChg0biOjLL78UW7p//0JGli1bxvN8eHj4UO7UbuS3bdsmtGdnZ0+dOnXt2rXt7e05OTlDWZKTFcra4Bxqenh4PPLIIykpKUT0l7/8pb29feDbdHV1nTp16scff6xQKPbs2VNXVzfwbYKTiouLQ0JCoqOjP/zwQ1wL5U4YzDGdNGkSETU3Nzc0NNxzzz299q+vr++1z5gxY/z9/a9fv15UVBQWFnbmzBnH/Q8ePOhktdCT3NzcuLi4pKSkJ598knUtI9ZgTq6UlJQQka+vb6+pmz17dmpqqpObFU7whvK052726aefxsTE/PWvf0Xq7qjBCZ7NZvv222//8Ic/aLVa4YBzsFy9erWiosLT0/O3v/3tIG6WiKxW66ZNm8aPH+/q6urr6xsXFye+nWZkZIin8leuXElISPDy8vLx8YmOjr506ZJ0I7W1tVu3bp04caJarQ4MDJw3b15qampLS0v3Dq6urgaDYcGCBdnZ2dItFBcXL168WK/X63S60NDQ255NOVlqSUlJfHy8j4+PsFhTU9PXMXnvvfcSExOzsrKio6MHsYyOjo60tLSIiAg/Pz+NRmMymd5+++2uri5xy21tbdu3b588ebJWq/X29l60aNHhw4els3TOcLCX+vr67tNvHR0dYsuSJUuGeKhv8ytj3WdBuhNOdu1MmjTpH//4hzM9iejDDz+8bU9xcuXWrVunT5+eNWuWq6vrRx99dNsyuk+uOKm8vHzcuHFGo/HIkSNNTU0//fSTxWJxc3PLy8sT+8TExAj15OXl2Wy2Y8eOaTSaGTNmiB0qKiqCgoL8/PwyMzMbGxsrKyuTkpKIaM+ePdIORqMxMzOzoaGhpKQkLi6O47j9+/cLHS5cuODl5RUQEHD06NGmpqazZ8/Onz9//PjxarW6H6VaLJbs7OybN28WFBS4uLhYrVae58PCwry9vfPz8x2MhjDy7u7uRPTss88OcMS6l5GZmUlEr776al1dndVqfeeddxQKxbZt28Qbrlu3Tq/XHz16tLm5ubKyUpg+yc7OtqvwtlN0ol73EhkZqVAoLl68KL1VcHDwgQMHBmWoHaBuvzI2oOCJA9He3n758uX//d//5TguLi7u1q1bPfUUzJo1q6fg2YmNjbUbKal+B0+43oA44jzPV1RUqNXqadOmiS3CEGdmZootwuuiOMpr167tPqBRUVFi8IQOH3/8sbi2tbV19OjRGo2msrKS5/mlS5cS0SeffCJ2uH79ulqtlgbP+VKzsrK631OLxWIwGKRPne6EkZ80aZLwOy47d+7s3mcgZWRmZs6ZM0faIlxsrKGhQVgMCgoKCQmRdrjvvvv6ETzHe/n666+JKDExUeyQk5MTEBAgPl0HONQO3KngiVasWEFEu3btctzTQfDEnmVlZcJnA88//3xPZfQ7eHq9XqFQiA+J4MEHHySia9euCYvCEAsJEWzZsoWIioqKxI0QUWNjo4O9dO8g/Pbz3//+d57nPTw8iKipqUnawWQySYPnfKk1NTVOD8CviCOfl5cnlLR79+7u92UQy9i5cycRiS8HGzduJKL169fn5+d3dHQ4qLBP98tuLzzPm0wmrVYrVhgTE/Paa6/dofso1T14g/yfK8LPPB4/ftxxt5ycHOHdwIGAgIDU1NSJEyfu3LlT/Jx9ULS1tTU0NHR1den1eumh/48//khEFy5ckHaWftgoXH9cOG0QNuLm5iY8U3vaS/cOwg8/V1ZWtrW1NTU1ubm5Ccd4olGjRvWvVJ1O15/hkAgODv7yyy/d3d2fffbZt956a7DKaGho2L59u8lkMhgMwg2FD3ubm5uFDsnJyR999NHly5fDw8M9PT2joqI+++yzvhbf616IaPPmzc3NzXv37iWi0tLSEydOiHNIQzzUgxw8IdzSuzoQbm5ur776Ks/zL7744qBsUKBWq728vJRKZXt7e/cXp7CwMCc3otfrW1tbm5qa+tShqqqKiPz8/NRqtYeHR2trq81mk3aQfmI5KKX2yaxZs7KysnQ63ZYtW959991BKWPRokVJSUnr168vLS3t6urieX7Pnj3072cLEXEct2rVqm+++aa+vj4jI4Pn+bi4uDfffLNPlfe6FyJasWKF0Wh877332tradu/evWbNGoPBMCj3sa8GOXjffvstEc2YMcOZztOnT+/1Y7elS5c+8MADx48fP3bs2CDU929xcXEdHR25ubnSxtdff33s2LEdHR1ObiQ2NpaIsrKypI0PPPCAcEQqdjhy5Ii4tq2t7fjx4xqNJjIykogWLFhARMI//QhqamqET2UGt9Q+CQ0NPXLkiFar3bRpU3Jy8gDL6OzszM3N9fPz27Rpk6+vL8dxRCSd+CUiLy+v4uJiIlKpVBEREcL8oXTcHFAqlcXFxc7shYjUanViYmJ1dfXu3bsPHDjwzDPPSNcO6VBLYz2QyZV//vOfwuRKQEBAeXl5Tz2lpk2bJp146Kmn8AA8+OCDwsuYVL/P8aqqqiZOnDhhwoSsrKz6+vra2tqUlBStVis9EBeO5ltaWsQW4T/XTp8+LSwKk5b+/v5ffPFFY2PjtWvXNm7caDQa//Wvf0k7CLOajY2N4qzmvn37hA4XL1709vYWZzV//vnnyMjIUaNGSc/x+leqyPlZTbuRP3HihEajISLhf2UHUsbcuXOJ6I033rBarc3NzSdOnBg7diwRHTt2TOig1+stFktRUVFra2tVVdWf//xnItqxY4fjCgUuLi7nz593Zi8Cq9Wq0Wg4juu+tQEOtQM0KJMr3Q9wOY7z8PAwm83PP/98VVWVg552xODZ9UxISJDuUfwn3VmzZgktH374od2m7KYoeiV8wjZhwgSVSuXr6zt//nzxEcrPz5du+aWXXuJ/fU2lhQsXCj1ramo2b94cFBSkUqn8/f2XLVtWWloq3Yu0g16vj4yMPH78uLRDSQLAHKgAABMBSURBVEnJ4sWLPT09hc8qvvjiC/F/NZ944om+ltr94QsNDXU8q2k38tIpzW+++UbIHhElJSX1uwyr1bphw4YxY8aoVCqj0bh27Vrx3EGYMDxz5syGDRvuv/9+4XO8mTNn7t+/X3yd7fVZJASv172I1q9fT0SnTp0ayLOi+1A7QN2C96vLdKWnpwvPeMf3E0DWPvzww+Tk5MGdsXOM47i0tDTpVUnwfTy466SkpGzdupVtDQge3BU++OCD2NhYm82WkpJy48YN5pfEGpnB43omnLjDXSgjI8NgMLz//vsHDx5k/l2nkflVK5ymgp1169atW7eOdRX/MTLf8QCGOQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+Agdt8O0H4oRgAuHN+9dMPZWVleXl5DKsB5wm/XSf+ohkMcyEhIYGBgeIih6+uyZTwHer09HTWhUB/4BwPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4ABBA+AAQQPgAEED4CB21yKGYanmpqaxsZGcfHmzZtEdPnyZbHF09PznnvuYVAZ9B2uCCsbf/3rX9etW+egwwcffPDEE08MWT0wEAiebNy4ccNoNLa3t992rUqlqqqqMhgMQ1wV9A/O8WTDYDBERUUplbc5O1AqlQsWLEDqZATBk5OVK1d2dnZ2b+/s7Fy5cuXQ1wP9hkNNOWltbfXx8WlubrZr12g0NTU1Wq2WSVXQD3jHkxM3N7fY2FiVSiVtVKlU//Vf/4XUyQuCJzOPPfaY3fxKe3v7Y489xqoe6B8caspMR0fHqFGjbty4IbZ4eXlVV1fbvQ3CMId3PJlRKpXLli1zdXUVFlUq1WOPPYbUyQ6CJz/Lly+/deuW8Hd7e/vy5cvZ1gP9gENN+eF5PjAwsLy8nIj8/PzKy8s5jmNdFPQN3vHkh+O4lStXurq6qlSq1atXI3VyhODJknC0iflM+ZLBtxOWLl3KuoThyN3dnYh27NjBupDh6NChQ6xL6IUMzvE4jps5c2ZgYCDrQoaX8+fPE9H999/PupDhpaysrKCgQAbPahmUyHFpaWnx8fGsCxleLl26REQTJ05kXcjwkp6enpCQMPyf1TI41ITbQuRkDZMrAAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADIzM4B08eJDjOI7j3NzcWNcydNzd3TkJhUJhMBjMZnNiYuIPP/zAujr4lZEZvGXLlvE8Hx4ezrqQIWWz2U6fPk1EMTExPM+3t7cXFxe/8sorxcXF06dP//3vf9/9J6iBlZEZPFlzd3efPXv2wLfj4uJiNBpjYmJOnDjx/PPPp6amLl++fPh/Uc3OYI3GcIPg3RVee+21hx9++PDhwwcPHmRdCxAheHcJjuOefvppItq7dy/rWoBoJAWvuLh48eLFer1ep9OFhobm5ORI12ZkZIizDiUlJfHx8T4+PsJiTU0NEdXW1m7dunXixImurq4Gg2HBggXZ2dnCbXft2iX0DAwMLCwsDA8P9/Dw0Gq1YWFhubm50r042MiOHTuEjYgHTl999ZXQIl4/WdjRzZs3c3NzhVW3vRpe/wj7LSgoaG9vx2iwxw97RJSWlua4z4ULF7y8vAICAo4ePdrU1HT27Nn58+ePHz9erVZLu8XExBCRxWLJzs6+efNmQUGBi4uL1WqtqKgICgoyGo2ZmZkNDQ0lJSVxcXEcx+3fv1+8rdls1ul0wcHBeXl5NputsLBwypQprq6uJ0+eFDo4sxGdTjdr1ixpSdOmTfPx8ZG2dO8jCAsL8/b2zs/PdzAO0skVOy0tLcIjXl5ePgJGoydpaWnyeFazLqB3zgRP+AnATz75RGy5fv26Wq2+bfCysrLsbr527Voi+vjjj8WW1tbW0aNHazSayspKocVsNhPR6dOnxT5nz54lIrPZ7PxGBvJUs1gsBoMhLy/PwTg4CJ44pWkXPJmORk/kErwRcqj51VdfEVFkZKTYMnr06Pvuu++2nR966CG7ls8++4yIFi5cKLao1erw8PCWlpavv/5abNTpdFOnThUXTSbT6NGji4qKKioqnN9Iv508ebKuri44OLh/NxeKVKlU4rGcQKajIXcjIXhtbW1NTU1ubm7Cb7yKRo0addv+Op3O7uYNDQ1ubm4eHh7SdqPRSESVlZVii5eXl92mhF1UV1c7vxFWhJPe4OBgu0sL3Z2jwdxICJ5arfbw8GhtbbXZbNL2uro6J2+u1+tbW1ubmpqk7VVVVUTk5+cnttTW1vK//hysurqaiEaNGuXkRhQKhXihH0F9fb1dPXfiWghdXV3JyclE9NRTTznueTeMxnAwEoJHRAsWLKB/H3AKampqSkpKnLx5bGwsER05ckRsaWtrO378uEajkR6+tra2FhYWiovnzp0rLy83m83+/v5ObsTf3//69etih8rKyqtXr9oVo9VqxafjpEmT9u3b5+S9cOBPf/rTd999Fxsb68zv4Y/40RgWWJ9k9o6cmFy5ePGit7e3OKv5888/R0ZGCi+90m7CdEJLS4vdzaVTcI2NjeIU3L59+8Q+ZrNZr9eHh4c7M4/X00aED9PefffdpqamixcvxsfHBwQE2E0nREVF6fX6q1ev5uXlKZXKX375RWjv66xmZ2dnVVVVRkbG3Llziejxxx9vbm4eMaPRE7lMrsihRCeCx/N8SUnJ4sWLPT09NRrNjBkzvvjiC/F/NZ944on8/HzHrzg1NTWbN28OCgpSqVR6vT4yMvL48ePSDmazOSAg4JdffomMjPTw8NBoNBaLJScnp08bqa+vX7dunb+/v0ajmT17dmFh4bRp04R6XnjhBaFPcXFxaGioTqcbM2ZMcnKyeNvQ0FDHs5p2Z2scx+n1epPJtHHjxh9++EHacwSMRk/kEjxcO8FZU6dOrampKSsrY1vGMDFsR0Mu104YIed4APKC4AEwgOD1TvinwaKiouvXr3Mc9/LLL7OuiCWMxqDAOR6MKDjHA4AeIXgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMyOPbCTNnzgwMDGRdCMhAWVlZQUGBDJ7Vw79EZ34Y6y70/fffE9H06dNZFzIcHTp0iHUJvZBB8OC2hC8opqensy4E+gPneAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAzgirCykZqa+tZbb3V2dgqLVquViHx9fYVFFxeXzZs3r127llV50CcInmyUlJRMnjzZQYfz58877gDDBw41ZWPSpEkmk4njuO6rOI4zmUxInYwgeHKyevVqFxeX7u1KpXLNmjVDXw/0Gw415aS8vDwwMLD7Q8Zx3NWrVwMDA5lUBf2Adzw5GT16dEhIiELxq0dNoVCEhIQgdfKC4MnMqlWr7E7zOI5bvXo1q3qgf3CoKTN1dXVGo7Gjo0NscXFxqaqq8vHxYVgV9BXe8WTG29s7IiJCqVQKiy4uLhEREUid7CB48rNy5cquri7hb57nV61axbYe6AccasrPzZs377nnntbWViJSq9U1NTXu7u6si4K+wTue/Oh0ukcffVSlUimVysWLFyN1coTgydKKFSs6Ojo6Ozsfe+wx1rVAfyhZF9AH+fn5165dY13FsNDZ2enm5sbzvM1mS09PZ13OsDBmzJjg4GDWVTiNl48lS5awHi0YvpYsWcL6GdoHcnrHI6IlS5YcOnSIdRXDQnZ2Nsdxc+bMYV3IsLB06VLWJfSNzIIHIovFwroE6D8ET67s/mMT5AUPHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AAwgeAAMIHgADCB4AA3dd8Hbt2sVxHMdxzH+I8vz58wkJCX5+fkqlUijJy8trKAtwd3fnJBQKhcFgMJvNiYmJP/zww1BWche664K3bds2nufNZvOQ7dFms917773R0dHSxitXrgQHB58/f/7TTz9tbGxsbGxMT08f4v97ttlsp0+fJqKYmBie59vb24uLi1955ZXi4uLp06f//ve/b25uHsp67ip3XfCGHs/zXV1d4u+CCfbt29fQ0JCcnBwSEqLVaj08PJYuXVpXV8eqSCJycXExGo0xMTEnTpx4/vnnU1NTly9fzuO3sO4MBO+O8/DwuHTpUlZWlrTxwoULRDRlyhRGRfXitddee/jhhw8fPnzw4EHWtYxMCB4b7e3tRKRWq1kXcnscxz399NNEtHfvXta1jEwjM3i1tbVbt26dOHGiWq0ODAycN29eampqS0tLT/07OjrS0tIiIiL8/Pw0Go3JZHr77belB4dtbW3bt2+fPHmyVqv19vZetGjR4cOHxYuzOlibkZEhzl4Iv4QptHz++edEpNFouF+zu6Sr1WrdtGnT+PHjXV1dfX194+Lizpw5I6ySbrmkpCQ+Pt7Hx0dYrKmpGfgYzp49m4gKCgqE1wjni7ly5UpCQoKXl5ePj090dPSlS5ecHEbHuxhp2P7kS58sWbLEmR+0qaioCAoK8vPzy8zMbGxsrKysTEpKIqI9e/aIfcxmc0BAgLiYmZlJRK+++mpdXZ3Van3nnXcUCoUwDSNYt26dXq8/evRoc3NzZWXltm3biCg7O9uZtTzPx8TEEFFLS4uDFuHSymvWrBFbysvLx40bZzQajxw50tTU9NNPP1ksFjc3t7y8PLvtWCyW7OzsmzdvFhQUuLi4WK1WnufDwsK8vb3z8/MdjJV0csWO+DpVXl7ep2JiYmLy8vJsNtuxY8c0Gs2MGTOcHEZndtETJ58bw8cIDJ7wppGWliZtjIqKchy8OXPmSPuvXLlSpVI1NDQIi0FBQSEhIdIO9913n/iMcbyW72/whGtNHjhwQGypqKhQq9XTpk2z205WVlb3cbBYLAaDwfGz1kHwxClNIXjOF5OZmSm2CD8MJ7wQ8L0NlDO76AmCdwc5Obh6vZ6IGhsbHfSxC153O3fuJCLxWbtx40YiWr9+fX5+fkdHh11nx2v5/gZPr9crFAox/IIHH3yQiK5duybdTk1NjYP74oCD4AmHiCqV6tatW30qprKyUuywZcsWIioqKhIWHQ+UM7voieyCN9LO8dra2hoaGtzc3Dw8PJy/VUNDw/bt200mk8FgEE5UnnvuOSISX/WTk5M/+uijy5cvh4eHe3p6RkVFffbZZ+LNHa8dyB3p6urS6/XSk8Aff/yR/j0pKtLpdAPcXXc5OTlEFBwcrFKp+lSM8MIncHV1JSLxbNnBQPVpFyPASAueWq3W6/Wtra1NTU3O32rRokVJSUnr168vLS3t6urieX7Pnj1ExP/7UyyO41atWvXNN9/U19dnZGTwPB8XF/fmm286s7bfd8TLy0upVLa3t3d/vQwLCxvIxnvV1dWVnJxMRE899dQgFuNgoNje36E30oJHRLGxsURk97nZAw88IBz2dNfZ2Zmbm+vn57dp0yZfX1/heqt2U6BeXl7FxcVEpFKpIiIihEm8I0eOOLO23+Li4jo6OnJzc6WNr7/++tixY6UXprwT/vSnP3333XexsbHiD8UOSjGOB4rh/R16IzB4//d//xcUFLRlyxZhcqysrCwxMbGioqKn4Lm4uMyZM6eysnLnzp01NTUtLS3Z2dkpKSl23f7whz+cPXu2ra2turr6jTfe4Hl+7ty5Tq7t9x2ZOHHi448//uWXXzY0NNTV1f3lL3955ZVXdu3aJV6Y0oG5c+f6+PgUFBQ4ubuurq7q6urPP/88PDz8jTfeePzxxw8cOCBe9nmAxYgcDNRg7UIeBu908Y5z/gS6pqZm8+bNQUFBKpXK399/2bJlpaWlwiph1kT00ksv8TxvtVo3bNgwZswYlUplNBrXrl374osvCh2EKbUzZ85s2LDh/vvvFz6Amjlz5v79+4WDUsdr7U72VqxY0b2F5/nIyEhp47fffitsWfhAcsKECSqVytfXd/78+ceOHRNW5efnO34oQ0NDHc9q2p0Zchyn1+tNJtPGjRt/+OGH7v2dL0YYVWnLwoULex1Gx7twTHaTK3K6MKVw2INrJ0B3sntujMBDTYDhD8EDYADBA2AAwQNgAMEDYADBA2AAwQNgAMEDYADBA2AAwQNgAMEDYADBA2AAwQNgAMEDYADBA2AAwQNgAMEDYEBmP2VRVlaWnp7OugoYdsrKyphf8LBPZBa8goKChIQE1lXAcCT8arVcyOk3VwBGDJzjATCA4AEwgOABMIDgATDw/wAgM/VCeLYisAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.utils.plot_model(classifier_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WbUWoZMwc302"
   },
   "source": [
    "## Model training\n",
    "\n",
    "You now have all the pieces to train a model, including the preprocessing module, BERT encoder, data, and classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WpJ3xcwDT56v"
   },
   "source": [
    "### Loss function\n",
    "\n",
    "Since this is a binary classification problem and the model outputs a probability (a single-unit layer), you'll use `losses.BinaryCrossentropy` loss function.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 4: define your loss and evaluation metric here. Since it is a binary classification use BinaryCrossentropy and BinaryAccuracy**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:23.893393Z",
     "iopub.status.busy": "2021-03-04T02:23:23.892636Z",
     "iopub.status.idle": "2021-03-04T02:23:23.898143Z",
     "shell.execute_reply": "2021-03-04T02:23:23.898618Z"
    },
    "id": "OWPOZE-L3AgE"
   },
   "outputs": [],
   "source": [
    "loss = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
    "metrics = tf.metrics.BinaryAccuracy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "77psrpfzbxtp"
   },
   "source": [
    "### Optimizer\n",
    "\n",
    "For fine-tuning, let's use the same optimizer that BERT was originally trained with: the \"Adaptive Moments\" (Adam). This optimizer minimizes the prediction loss and does regularization by weight decay (not using moments), which is also known as [AdamW](https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW).\n",
    "\n",
    "In past labs, we have been using the Adam optimizer which is a popular choice. However, for this lab we will be using a new optimizier which is meant to improve generalization. The intuition and algoritm behind AdamW can be found in this paper [here](https://arxiv.org/abs/1711.05101).\n",
    "\n",
    "For the learning rate (`init_lr`), we use the same schedule as BERT pre-training: linear decay of a notional initial learning rate, prefixed with a linear warm-up phase over the first 10% of training steps (`num_warmup_steps`). In line with the BERT paper, the initial learning rate is smaller for fine-tuning (best of 5e-5, 3e-5, 2e-5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:23.904944Z",
     "iopub.status.busy": "2021-03-04T02:23:23.903782Z",
     "iopub.status.idle": "2021-03-04T02:23:23.906910Z",
     "shell.execute_reply": "2021-03-04T02:23:23.907325Z"
    },
    "id": "P9eP2y9dbw32"
   },
   "outputs": [],
   "source": [
    "epochs = 5\n",
    "steps_per_epoch = tf.data.experimental.cardinality(train_ds).numpy()\n",
    "num_train_steps = steps_per_epoch * epochs\n",
    "num_warmup_steps = int(0.1*num_train_steps)\n",
    "\n",
    "init_lr = 3e-5\n",
    "optimizer = optimization.create_optimizer(init_lr=init_lr,\n",
    "                                          num_train_steps=num_train_steps,\n",
    "                                          num_warmup_steps=num_warmup_steps,\n",
    "                                          optimizer_type='adamw')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SqlarlpC_v0g"
   },
   "source": [
    "### Loading the BERT model and training\n",
    "\n",
    "Using the `classifier_model` you created earlier, you can compile the model with the loss, metric and optimizer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 5: complile the model using the optimizer, loss and metrics you defined above**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:23.922353Z",
     "iopub.status.busy": "2021-03-04T02:23:23.921618Z",
     "iopub.status.idle": "2021-03-04T02:23:23.934573Z",
     "shell.execute_reply": "2021-03-04T02:23:23.933739Z"
    },
    "id": "-7GPDhR98jsD"
   },
   "outputs": [],
   "source": [
    "classifier_model.compile(optimizer=optimizer,\n",
    "                         loss=loss,\n",
    "                         metrics=metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CpBuV5j2cS_b"
   },
   "source": [
    "Note: training time will vary depending on the complexity of the BERT model you have selected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 6: write code to fit the model and start training**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:23:23.940999Z",
     "iopub.status.busy": "2021-03-04T02:23:23.940247Z",
     "iopub.status.idle": "2021-03-04T02:30:58.712991Z",
     "shell.execute_reply": "2021-03-04T02:30:58.713410Z"
    },
    "id": "HtfDFAnN_Neu"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model with https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/1\n",
      "Epoch 1/5\n",
      "625/625 [==============================] - 277s 434ms/step - loss: 0.5724 - binary_accuracy: 0.6751 - val_loss: 0.3807 - val_binary_accuracy: 0.8368\n",
      "Epoch 2/5\n",
      "625/625 [==============================] - 269s 431ms/step - loss: 0.3593 - binary_accuracy: 0.8374 - val_loss: 0.3838 - val_binary_accuracy: 0.8472\n",
      "Epoch 3/5\n",
      "625/625 [==============================] - 269s 430ms/step - loss: 0.2732 - binary_accuracy: 0.8813 - val_loss: 0.4125 - val_binary_accuracy: 0.8400\n",
      "Epoch 4/5\n",
      "625/625 [==============================] - 268s 428ms/step - loss: 0.2042 - binary_accuracy: 0.9166 - val_loss: 0.4380 - val_binary_accuracy: 0.8546\n",
      "Epoch 5/5\n",
      "625/625 [==============================] - 266s 426ms/step - loss: 0.1599 - binary_accuracy: 0.9402 - val_loss: 0.4776 - val_binary_accuracy: 0.8504\n"
     ]
    }
   ],
   "source": [
    "print(f'Training model with {tfhub_handle_encoder}')\n",
    "history = classifier_model.fit(x=train_ds,\n",
    "                               validation_data=val_ds,\n",
    "                               epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uBthMlTSV8kn"
   },
   "source": [
    "### Evaluate the model\n",
    "\n",
    "Let's see how the model performs. Two values will be returned. Loss (a number which represents the error, lower values are better), and accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:30:58.718690Z",
     "iopub.status.busy": "2021-03-04T02:30:58.717848Z",
     "iopub.status.idle": "2021-03-04T02:32:02.041855Z",
     "shell.execute_reply": "2021-03-04T02:32:02.041151Z"
    },
    "id": "slqB-urBV9sP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 124s 159ms/step - loss: 0.4560 - binary_accuracy: 0.8569\n",
      "Loss: 0.4560081660747528\n",
      "Accuracy: 0.8569200038909912\n"
     ]
    }
   ],
   "source": [
    "loss, accuracy = classifier_model.evaluate(test_ds)\n",
    "\n",
    "print(f'Loss: {loss}')\n",
    "print(f'Accuracy: {accuracy}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uttWpgmSfzq9"
   },
   "source": [
    "### Plot the accuracy and loss over time\n",
    "\n",
    "Based on the `History` object returned by `model.fit()`. You can plot the training and validation loss for comparison, as well as the training and validation accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:32:02.067427Z",
     "iopub.status.busy": "2021-03-04T02:32:02.063402Z",
     "iopub.status.idle": "2021-03-04T02:32:02.384026Z",
     "shell.execute_reply": "2021-03-04T02:32:02.384509Z"
    },
    "id": "fiythcODf0xo"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'binary_accuracy', 'val_loss', 'val_binary_accuracy'])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f2a3b809950>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAGDCAYAAABuj7cYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABjtklEQVR4nO3deXiU5bnH8e9NWAKERfYlIKggYlmNoKCI+1q34kG0KtK6b+hRa21V1NrjaTnVWle0rtVSWyu11hUVcZdFQBFEQERkEUEg7Fnu88czIZNkkgyQyUxmfp/rmivzrnM/eQm586zm7oiIiIhIaqiX7ABEREREpJSSMxEREZEUouRMREREJIUoORMRERFJIUrORERERFKIkjMRERGRFKLkTESqZWYvm9l5NX1uMpnZEjM7KgH3dTPbJ/L+QTO7KZ5zd+Fzzjaz13Y1ziruO9zMltX0fUUkfvWTHYCIJIaZbYzabAJsA4oi2xe5+9Px3svdj0/EuenO3S+uifuYWTfgK6CBuxdG7v00EPczFJG6Q8mZSJpy95yS92a2BPi5u08uf56Z1S/5hS8iIsmnZk2RDFPSbGVmvzCzlcBjZraHmb1oZqvN7IfI+9yoa6aY2c8j70eb2btmNj5y7ldmdvwuntvdzKaaWb6ZTTaz+8zsL5XEHU+Mt5vZe5H7vWZmbaKOn2NmX5vZGjP7VRXfn4PMbKWZZUXtO83M5kTeDzKzD8xsnZmtMLN7zaxhJfd63Mx+E7V9XeSa5WY2pty5J5rZJ2a2wcy+MbNxUYenRr6uM7ONZnZwyfc26vohZjbNzNZHvg6J93tTFTPbL3L9OjOba2YnRx07wcw+j9zzWzO7NrK/TeT5rDOztWb2jpnp941InPTDIpKZOgCtgD2BCwn/FzwW2e4KbAHureL6wcAXQBvgd8Cfzcx24dxngI+B1sA44JwqPjOeGM8CzgfaAQ2BkmShN/BA5P6dIp+XSwzu/iGwCTii3H2fibwvAq6OlOdg4Ejg0iriJhLDcZF4jgZ6AOX7u20CzgVaAicCl5jZqZFjwyJfW7p7jrt/UO7erYD/APdEyvYH4D9m1rpcGSp8b6qJuQHwb+C1yHVXAE+b2b6RU/5MaCJvBvwIeDOy/7+BZUBboD1wI6C1AkXipORMJDMVA7e4+zZ33+Lua9z9OXff7O75wB3AYVVc/7W7P+zuRcATQEfCL+G4zzWzrsCBwM3uvt3d3wVeqOwD44zxMXdf4O5bgGeB/pH9I4AX3X2qu28Dbop8DyrzV2AUgJk1A06I7MPdZ7j7h+5e6O5LgIdixBHLf0Xi+8zdNxGS0ejyTXH3T9292N3nRD4vnvtCSOa+dPenInH9FZgP/DjqnMq+N1U5CMgB7ow8ozeBF4l8b4ACoLeZNXf3H9x9ZtT+jsCe7l7g7u+4FnIWiZuSM5HMtNrdt5ZsmFkTM3so0uy3gdCM1jK6aa+clSVv3H1z5G3OTp7bCVgbtQ/gm8oCjjPGlVHvN0fF1Cn63pHkaE1ln0WoJTvdzBoBpwMz3f3rSBw9I012KyNx/JZQi1adMjEAX5cr32AzeyvSbLseuDjO+5bc++ty+74GOkdtV/a9qTZmd49OZKPv+xNC4vq1mb1tZgdH9v8eWAi8ZmaLzeyG+IohIqDkTCRTla/F+G9gX2CwuzentBmtsqbKmrACaGVmTaL2dani/N2JcUX0vSOf2bqyk939c0IScjxlmzQhNI/OB3pE4rhxV2IgNM1Ge4ZQc9jF3VsAD0bdt7pap+WE5t5oXYFv44iruvt2KddfbMd93X2au59CaPKcRKiRw93z3f2/3X0vQu3dNWZ25G7GIpIxlJyJCEAzQh+udZH+S7ck+gMjNVHTgXFm1jBS6/LjKi7ZnRj/AZxkZodEOu/fRvX//z0DXElIAv9eLo4NwEYz6wVcEmcMzwKjzax3JDksH38zQk3iVjMbREgKS6wmNMPuVcm9XwJ6mtlZZlbfzEYCvQlNkLvjI0JfuOvNrIGZDSc8o4mRZ3a2mbVw9wLC96QIwMxOMrN9In0LS/YXxfwEEalAyZmIANwNNAa+Bz4EXqmlzz2b0Kl+DfAb4G+E+dhiuZtdjNHd5wKXERKuFcAPhA7rVfkrMBx4092/j9p/LSFxygcejsQcTwwvR8rwJqHJ781yp1wK3GZm+cDNRGqhItduJvSxey8yAvKgcvdeA5xEqF1cA1wPnFQu7p3m7tuBkwk1iN8D9wPnuvv8yCnnAEsizbsXAz+N7O8BTAY2Ah8A97v7lN2JRSSTmPpoikiqMLO/AfPdPeE1dyIiqUo1ZyKSNGZ2oJntbWb1IlNNnELouyQikrG0QoCIJFMH4J+EzvnLgEvc/ZPkhiQiklxq1hQRERFJIUlp1jSz48zsCzNbGGv+GwvLy6w3s1mR183JiFNERESkttV6s2Zkwsj7CEuYLAOmmdkLkXmFor3j7ifVdnwiIiIiyZSMPmeDgIXuvhjAzCYSOgGXT852Wps2bbxbt267exsRERGRhJsxY8b37t62/P5kJGedKbuEyTLCwsjlHWxmswkzVF8bmaeoSt26dWP69Ok1E6WIiIhIAplZ+WXXgOQkZ7GWOSk/KmEmYcHcjWZ2AmFofY+YNzO7ELgQoGvX8quhiIiIiNQtyRgQsIyy68vlEmrHdnD3De6+MfL+JaCBmcVcANjdJ7h7nrvntW1boWZQREREpE5JRnI2DehhZt0ja9ydSVjsdwcz6xBZk43IGnP1CEuSiIiIiKS1Wm/WdPdCM7sceBXIAh5197lmdnHk+IPACOASMyskLHR8pqfChGyTJkH37tCvX7IjERERkTSVVpPQ5uXlecIGBBQXw777wsKFMHIk3Hpr2BYRERHZBWY2w93zyu/X2prxqlcPPv4YbrwRXnwReveGMWNgyZJkRyYiIiJpRMnZzthjD7jjDli8GK68Ep55Bnr2hMsvhxUrkh2diIiIpAElZ7uiXTu4667QxHn++fDQQ7D33nD99bBG4xZERERk1yk52x25uSExmz8fRoyA8ePDgIFx42DDhmRHJyIiInWQkrOasPfe8OST8OmncPTRYbBA9+7wu9/B5s3Jjk5ERETikJ8Pn38OU6cmNw6N1kyEGTPg17+GV16BDh3gV7+CCy6ARo2SHZmIiEhGKiiAb7+Fb76BpUvDq+R9ydd168K52dmhbsVirWlUgyobrankLJHefTckZlOnQteucMstcO65UD8Zq2aJiIikJ3f4/vuyiVb55Gv58nBetFatoEuX8Cu65GvJ+yFDwkQNiaTkLFnc4fXXQ5I2fXoY3XnrrfBf/5X4py4iIpIGNm2qusbrm29g69ay1zRqFDvpKvnapQvk5CSnPCWUnCWbO/zrX3DTTfDZZ9C3L9x+O/z4x4mvNxUREUlRhYWhVitW0lXyfu3asteYQceOsZOukn1t2qT+r1clZ6miqAj+9rfQxLlwIQweDL/5DRx5ZOr/KxIREdkJ7mGGqcqSrqVLQ2JWXFz2upYtq6716twZGjRISpFqlJKzVFNQAE88AbfdFv6FDh8eJrgdMiTZkYmIiMRl8+bwK6yqJsctW8pe07Bh5UlXyddmzZJTntqm5CxVbd0KEyaExOy77+CEE0JN2oAByY5MREQyWFFRWPymsqRr6dKK866bhUkKKmtq7NIF2rZVl+sSSs5S3aZN8Kc/hbnRfvghTGp7222w337JjkxERNKMe/hVU1U/r2+/DQlatBYtKk+6unYNzY0NGyanTHWRkrO6Yt06+MMfwvJQmzfDT38aVhzo3j3ZkYmISB2xZQssW1b11BKbNpW9pkGD0lGMlY1ubNEiOeVJV0rO6prVq+F//xfuuy8MZfn5z8PEtp07JzsyERFJoqIiWLmy6n5eq1dXvK59+6pHN7Zrp+bG2qbkrK769tvQH+3hh8PktZdeCjfcEBrtRUQkrbjD+vVV9/P69tvwN3u0nBzYc8/Kmxxzc7VITSpSclbXffVVmLz2qaegSRMYOxb++7/DeGMREakTtm4NzY1VTS2xcWPZa+rXD8lVVVNLtGih2ZjqIiVn6WLevDBH2t//DnvsAdddB1deCU2bJjsyEZGMVlwMq1ZV3cl+1aqK17VrV/XUEu3bQ1ZW7ZdHEk/JWbr55JOw2sB//hN+sm+8ES66KKzWKiIiNW79+qprvJYtC1NYRmvatOp+Xrm5+m87kyk5S1fvvx8GCrz1VviJv+kmGD06PaZOFhGpZT/8AHPmhNfcuWWTrw0byp6blRWSq6qmlmjZUs2NUjklZ+nujTfC4uoffQT77BP6p515pobeiIjEUFQEixbB7NnhNWdO+Lp0aek5e+wRZjGqrMmxY0c1N8ruUXKWCdzhxRdDTdqcOfCjH4XF1U85RX+6iUjGWr8ePv20NBGbPRs++yxMJQkhwdp3X+jXr/TVt29IvvRfpySSkrNMUlwcBgzcfDMsWAB5eWFJqGOO0f80IpK2ioth8eKyNWGzZ8OSJaXn7LFH2SSsXz/o3Vv9viQ5lJxlosLCMPXGrbfC11/DsGFhzrRDDkl2ZCIiuyU/v2xt2Jw5YbtkGop69aBnz7I1Yf36hXm89TeqpAolZ5ls2zZ45JFQe7ZyJRx3XHh/wAHJjkxEpErFxaHmK7ombPbsUENWomXL0uQrujasSZNkRS0SHyVnEjpY3Hcf3HknrF0Lp58eFlfff/9kRyYiwqZNFWvD5swJtWQQarx69ChbE9avX+icr9owqYuUnEmpDRvCwur/93+hDeDss8Pi6nvvnezIRCQDuIeeFuVrwxYtCscAmjcvm4D17RvGOGm+bUknSs6kojVr4He/gz/9KcycOGZMmCctNzfZkYlImti8OYyMjO6kP2dOGEFZYp99KjZL7rmnasMk/Sk5k8qtWAG//S089FDoRXvJJfDLX4aVB0RE4uAeZsiPrgmbMwe+/DL0G4OwOHdJElbytU+fsF8kEyk5k+p9/XXog/b449C4MVx1FVx7bRh7LiISsWVLmD0/ullyzpwwu36J7t0rzhvWvbvmxRaJpuRM4rdgQVhcfeLEMAzq2mtDoqY/b0UyijssX15xFv0vviitDWvaNNR+RTdL9ukT+oyJSNWUnMnOmzMn9EF74QVo2zY0dV5yiWZrFElDW7fCvHkVmyXXrCk9Z889K84btvfeqg0T2VUplZyZ2XHAH4Es4BF3v7OS8w4EPgRGuvs/qruvkrME+eijsCTU5MlhBsebbgqDB7S4ukid4x6mOyw/i/78+WG9SQi9GmLVhrVsmdTQRdJOyiRnZpYFLACOBpYB04BR7v55jPNeB7YCjyo5SwFTpoTF1d9/H/baK0y/cdZZWvlXJEVt3162NqwkGVu9uvScLl0q1obts49+rEVqQ2XJWf0kxDIIWOjuiwHMbCJwCvB5ufOuAJ4DDqzd8Cq3YUPodpWxVfjDh8O778LLL4ck7dxzw4S2t90WJrTVuHeRpFm1qmJt2Lx5YRU3gEaNwjxhP/5x2dqwVq2SG7eIVJSM5Kwz8E3U9jJgcPQJZtYZOA04gmqSMzO7ELgQoGvXrjUaaHlHHBH+w+vQATp2DK9OnWK/b9cuTf/yNIMTTghLQD33XFhcfcQIGDgwLAl13HFK0kQSqKAgNEGWT8RWrSo9p3PnkHydeGJpItajB9RPxv/4IrLTkvGjGus3d/m21buBX7h7kVXzi97dJwATIDRr1kSAlbn88jCQccWKMILpq69CC9/331c8t149aN++6gSuU6dwTp38D7NePTjjjFBj9vTToYnzhBPCouq/+Q0cdliyIxSp877/vmIH/c8/D82VAA0bhtXXjj++tFmyb19o0ya5cYvI7klGWrAM6BK1nQssL3dOHjAxkpi1AU4ws0J3n1QrEVZi9OjY+7dvDx1sV6woTdyi3y9bBtOmwXfflS5NUsIsDISsKoHr2DHU1jVsmPAi7rysrNC8eeaZ8OijcPvtofnz6KPhjjvgwJRplRZJWYWFYXqK8ssZrVhRek7HjiHxOuaY0tqwnj01LkckHSVjQEB9woCAI4FvCQMCznL3uZWc/zjwYjoMCCgoCAlarAQu+v2qVaVzCEVr06Y0aauqRi6pM11s2QIPPAD/8z/hz/5TTgkJW58+SQxKJHWsXVuxg/7cubBtWzjeoAH07l1xce+2bZMbt4jUvJQZEODuhWZ2OfAqYSqNR919rpldHDn+YG3HVFsaNAh9QTp3rvq8oqIwmqqyBG7FitC0sXJlaWffaHvsUX0C17FjghYQbtwYrrkGLrgA7r4bxo8Pv1nOPBNuvTV0fBHJAEVFYemi8s2Sy5aVntOuXfjxuOKK0mSsV68UrSUXkVqjSWjrsOLiMEFkVbVwJe9L+qhEa968+j5xHTtCs2a7EeTatSFB++MfQ9XA6NFhEEGCB2+I1KYffii7qPfs2WGx761bw/H69WG//SrWhrVvn9y4RSS5Umaes0TKtOQsXu4hR6ougVu+vPSXSbSmTauugSvZbtGiioGaq1aFxdUfjFSMXnQR3Hhj6EwnUkcUFcGiRRVrw5YuLT2nTZuKa0rut1+YykJEJJqSM6mWO6xfH18St2lTxeuzs6uvhetYtIxWf7oVe/yx8Nvqyivhuus02ZLUKvdQm7x5c/Wv9etDN4KS2rDNm8M9srJCE2R0TVi/fuHvDc0mIyLxUHImNSo/v/oEbsWKMHFveQ0bQse2BXTcuoROa+bQscEaOh66D51GHEzH7o13JHOtW2fwhL8ZrKCgNDHasiW+BKq6V6z7xBp0U5lWrSrOot+7t5aZFZHdkzIDAiQ9NGsWXj17Vn3epk1lBzKUJnANWLGiB/O/6sJbSwv44c1m8GbZa+vXD7UQ1TWptm2bphP+ppiioviTpd1JqmINcqlOgwbQpEnZV+PG4Wv79hWPxftq3DisCtK6tWrDRKT2KDmThGraNKzTt88+lZ2RDWSz5Z3prLzxHla8u5AVLXuz/OjzWNF9CCu+y2L5cli8GN57L/aEv1lZYdRbdU2qdXbC32oUF4e+gomsZdq8uXSqh51Rr17liU+rVpCbW3lStTMJlOb6EpF0omZNSS1Tp4Z1O999F7p1CysPnH32jqwqesLfqqYaqWzC33btqp9mpKYm/N2Zfk27k1ht2bJr8e1OQlRVohS93bChapxERCqjPmdSd7jDq6/Cr38NM2aEXte33hrW8IyzE1pNTfgbvUKD2c4nVDvTr6lEo0a7ngzF+8rOVtIkIpJsSs6k7nGH55+Hm24Kw+X69w/rdp5wQo1lFtVN+FvyfuXK8JGJrGUq2af+cyIimUHJmdRdRUXw17/CLbeEzmcHHxzW7Tz88FoLwV01TSIiUrMqS840UYGkvqws+OlPYf58eOgh+OYbOOIIOOoo+PDDWglBiZmIiNQWJWdSdzRoABdeGBYsvOuuMDX7wQfDySeHGUJFRETSgJIzqXuys2Hs2NDEeccd8M47oT/amWfCF18kOzoREZHdouRM6q6cnLA+51dfhek3XnwxTNs+ZgwsWZLs6ERERHaJkjOp+1q2DKM4Fy+Gq66CZ54JSxdcfnkYaikiIlKHKDmT9NGuHfzhD7BwYag9e+gh2HtvuP56WLMm2dGJiIjERcmZpJ/cXHjwwTC6c8QIGD8euncPqw3EWoldREQkhSg5k/S1997w5JPw2WdwzDFhlYHu3eF3vwvT94uIiKSg3U7OzKypmdWLvO9pZiebmZYhltTRuzf84x9hKajBg+EXvwiJ27337tpq3iIiIglUEzVnU4FsM+sMvAGcDzxeA/cVqVkDB8JLL4WpN/bdF664IgwcePRRKCxMdnQiIiJAzSRn5u6bgdOBP7n7aUDvGrivSGIccgi89Ra89hq0bw8/+xnsvz9MnLhrK5WLiIjUoBpJzszsYOBs4D+RffVr4L4iiWMGRx8NH30EkyZBw4YwahQMGAAvvBAW0xQREUmCmkjOxgK/BJ5397lmthfwVg3cVyTxzOCUU8LyT888A1u2hO2DD4bJk5WkiYhIrdvt5Mzd33b3k939fyMDA7539ytrIDaR2lOvXqg5+/xzeOQRWL481KwdcQS8/36yoxMRkQxSE6M1nzGz5mbWFPgc+MLMrtv90ESSoH790Aftyy/hnntg3jwYOhROPBH++U/47rtkRygiImmuJpo1e7v7BuBU4CWgK3BODdxXJHkaNQqjORctgjvvhA8/hJ/8JAwg2G8/uPBC+Mtf4Ouvkx2piIikmZpIzhpE5jU7FfiXuxcA6qgj6aFp0zAv2ooVoXnzzjvDHGnPPgvnnAPdukHXrvDTn4bloj7/XP3URERkt9TEqMqHgCXAbGCqme0JaI0cSS8NG4ZBAgcfHJK1oqKw8sA778DUqfDGG/D00+HcNm3CdB3DhsGhh0L//qG5VEREJA7mCfgr38zqu3utz+qZl5fn06dPr+2PFQm1ZYsWhUStJGFbvDgcy8mBIUNCojZsGAwaBNnZyY1XRESSzsxmuHtehf27m5yZWQvgFmBYZNfbwG3uvn63brwLlJxJSvn225Colbw+/TTsb9gwJGiHHhpeQ4ZAixbJjVVERGpdIpOz54DPgCciu84B+rn76bt1412g5ExS2tq18N57pTVrM2aEZaPq1YN+/UqbQQ89FNq1S3a0IiKSYIlMzma5e//q9tUGJWdSp2zaFEaBliRrH34YJsGFsPZnSTPooYfCnnuGCXNFRCRtVJac1UQv5S1mdoi7vxv5oKHAlhq4r0h6a9oUjjwyvAC2b4eZM0v7rf3jH2FCXIDc3NJEbdiwMJ2HkjURkbRUEzVn/YAngZJOMz8A57n7nCquOQ74I5AFPOLud5Y7fgpwO1AMFAJjS5K/qqjmTNJKcXHZEaHvvBOm9ABo3bq0CfTQQ8OaoBoRKiJSpySsWTPqA5oDuPsGMxvr7ndXcl4WsAA4GlgGTANGufvnUefkAJvc3c2sL/Csu/eqLgYlZ5LWSkaElgwwmDo1bEMYEXrwwWVHhDZunNx4RUSkSols1gRCUha1eQ1wdyWnDgIWuvviSGATgVMISz+V3Gtj1PlN0aS2IqEZc599wuv888O+5cvLjgi95ZaQxDVsCAceWJqsaUSoiEidkah2kKo6w3QGvonaXgYMrnADs9OA/wHaASdW+kFmFwIXAnTt2nVXYhWpuzp1gpEjwwvghx/CiNCSZtDx48OqBvXqQd++ZUeEtm+f3NhFRCSmRE1Cu9TdY2ZKZnYGcKy7/zyyfQ4wyN2vqOT8YcDN7n5UdZ+rZk2RcjZtgo8+Km0G/eCD0hGhPXuWHRHarZsGGYiI1KIab9Y0s3xiNzcaUFVnl2VAl6jtXGB5ZSe7+1Qz29vM2rj797sUrEimatoUjjgivAAKCsqOCH3uOfjzn8Ox3Nyyydp++4UaNxERqVUJqTmr8gPN6hMGBBwJfEsYEHCWu8+NOmcfYFFkQMBA4N9ArlcTrGrORHZScTHMnVt2ROjyyN9KrVpVHBHaoEFy4xURSSMJHxAQL3cvNLPLgVcJU2k86u5zzeziyPEHgZ8A55pZAWHOtJHVJWYisgvq1YM+fcLr0kvDYILFi8uOCP3Xv8K5TZuGEaElNWuDB2tEqIhIAtR6zVkiqeZMJAFWrCg7InTOnJDENWhQdkTo0KEaESoishMSPs9ZKlByJlIL1q0rOyJ0+vTQl80srBEa3RTaoUOyoxURSVlKzkQkMTZvrjgidPPmcKxHj7LTd3TvrhGhIiIRSs5EpHYUFMAnn5TWrL3zTph/DaBz57IjQnv31ohQEclYSs5EJDmKi+Hzz8sma99+G461agWHHFKasGlEqIhkECVnIpIa3OGrr8qOCP3yy3CsSZOw1FRJM+jgwWGfiEgaUnImIqlr5cqyyVr0iNC8vNJm0KFDoWXLZEcrIlIjlJyJSN2xbh28/35pU+i0aaUjQvv2LdtvTSNCRaSOUnImInXXli0VR4Ru2hSO7bNPaaI2bJhGhIpInaHkTETSR8mI0OjJcdeuDcc6dSpbs7b//hoRKiIpScmZiKSv4mKYN6+0GXTq1NIRoXvsEUaEliRrAwdqRKiIpAQlZyKSOdxhyZKyC7ovWBCONWkS1ggtGRF60EEaESoiSaHkTEQy26pVZUeEzp5dOiL0gAPKjgjdY49kRysiGUDJmYhItPXrK44I3b49DCbo06dsv7WOHZMdrYikISVnIiJV2bIFPv64tGbt/fdLR4R27BgWdS959e8f1g2tXz+pIYtI3VZZcqb/WUREABo3hsMOCy+AwsIwIvS992DWrNAM+sYbYaQoQHY2/OhHpclav35hDrYWLZJVAhFJE6o5ExGJ1/btMH9+SNRmzy5N2r7/vvScbt1Kk7WSl+ZeE5EY1KwpIpII7rBiRWmiVvJasCBM8QHQvHmoVYtuFv3Rj0JtnYhkLDVriogkglmY+LZTJzjhhNL9mzfDZ5+VrWV78knIzw/H69WDnj3LNov26xf6t6mWTSSjKTkTEUmEJk1g0KDwKlFcHOZfi24S/egj+NvfSs9p06Zssta/P/TqpYlzRTKImjVFRJJt3TqYM6dss+inn8K2beF4w4bQu3fFWrZWrZIZtYjsJvU5ExGpSwoLQ7+16Fq22bNh5crSc3JzKw4+2GcfrSUqUkdkbJ+zgoICli1bxtatW5MdilQjOzub3NxcGqj5RiTModa7d3iNGlW6f9WqsjVss2fDyy9DUVE43rRpmEQ3ulm0Tx/IyUlKMURk56V9zdlXX31Fs2bNaN26NaZOtinL3VmzZg35+fl079492eGI1C1bt8Lnn1ec4mPdunDcDPbeu2KzaJcuGnwgkkQZW3O2detWunXrpsQsxZkZrVu3ZvXq1ckORaTuyc6GgQPDq4Q7fPNNxWbR554rPWePPSqufNC7NzRqVNslEJEoaZ+cAUrM6gg9J5EaZAZdu4bXj39cuj8/Pww2iK5le/jhMPUHhObUXr0q1rK1a5eMUohkpIxIzpJlzZo1HHnkkQCsXLmSrKws2rZtC8DHH39Mw4YNK712+vTpPPnkk9xzzz1VfsaQIUN4//33dzvWKVOmMH78eF588cXdvpeIpLBmzWDIkPAqUVQEixaVrWV7+214+unSczp0qDj4oGdPrS8qkgD6qUqg1q1bM2vWLADGjRtHTk4O11577Y7jhYWF1K/kP7a8vDzy8io0Q1dQE4mZiGS4rKyQaPXsCWecUbp/zZqyAw9mzap8fdGSmjatLyqy25Sc1bLRo0fTqlUrPvnkEwYOHMjIkSMZO3YsW7ZsoXHjxjz22GPsu+++ZWqyxo0bx9KlS1m8eDFLly5l7NixXHnllQDk5OSwceNGpkyZwrhx42jTpg2fffYZBxxwAH/5y18wM1566SWuueYa2rRpw8CBA1m8eHGVNWRr165lzJgxLF68mCZNmjBhwgT69u3L22+/zVVXXQWEJsipU6eyceNGRo4cyYYNGygsLOSBBx7g0EMPrZXvpYgkWOvWcMQR4VUien3Rklq2f/0L/vzn0nO6davYLKr1RUXillnJ2dix4T+TmtS/P9x9905dsmDBAiZPnkxWVhYbNmxg6tSp1K9fn8mTJ3PjjTfyXHSH3Yj58+fz1ltvkZ+fz7777ssll1xSYcqJTz75hLlz59KpUyeGDh3Ke++9R15eHhdddBFTp06le/fujIoekl+JW265hQEDBjBp0iTefPNNzj33XGbNmsX48eO57777GDp0KBs3biQ7O5sJEyZw7LHH8qtf/YqioiI2l/RbEZH01LBhqB3r2xfOOSfsq2x90X//W+uLiuyCzErOUsQZZ5xBVlYWAOvXr+e8887jyy+/xMwoKGkuKOfEE0+kUaNGNGrUiHbt2rFq1Spyc3PLnDNo0KAd+/r378+SJUvIyclhr7322jE9xahRo5gwYUKV8b377rs7EsQjjjiCNWvWsH79eoYOHco111zD2Wefzemnn05ubi4HHnggY8aMoaCggFNPPZX+/fvvzrdGROoirS8qUqMyKznbyRquRGnatOmO9zfddBOHH344zz//PEuWLGH48OExr2kUNbQ9KyuLwsLCuM7ZlXnsYl1jZtxwww2ceOKJvPTSSxx00EFMnjyZYcOGMXXqVP7zn/9wzjnncN1113Huuefu9GeKSBqqqfVF+/WD/fbT+qKSMZKSnJnZccAfgSzgEXe/s9zxs4FfRDY3Ape4++zajbJ2rF+/ns6dOwPw+OOP1/j9e/XqxeLFi1myZAndunXjb9H/AVZi2LBhPP3009x0001MmTKFNm3a0Lx5cxYtWkSfPn3o06cPH3zwAfPnz6dx48Z07tyZCy64gE2bNjFz5kwlZyJSuXr1YK+9wuu000r3l19fdNYsuPfe2OuLRte0aX1RSUO1npyZWRZwH3A0sAyYZmYvuPvnUad9BRzm7j+Y2fHABGBwbcdaG66//nrOO+88/vCHP3BEdKfbGtK4cWPuv/9+jjvuONq0acOg6L9gKzFu3DjOP/98+vbtS5MmTXjiiScAuPvuu3nrrbfIysqid+/eHH/88UycOJHf//73NGjQgJycHJ588skaL4OIZICWLWHYsPAqEWt90Vdfhcj/SUBYX7R8s6jWF5U6rtaXbzKzg4Fx7n5sZPuXAO7+P5Wcvwfwmbt3ru7esZZvmjdvHvvtt99ux12Xbdy4kZycHNydyy67jB49enD11VcnO6yY9LxEpFrl1xedNSuMIC1ZX7RJk7KDD/r1C9taX1RSTCot39QZ+CZqexlV14r9DHg5oRGluYcffpgnnniC7du3M2DAAC666KJkhyQisuvat4djjgmvEtHri5bUsv3tb/DQQ+F49Pqi0TVtWl9UUlAykrNYPwUxq+/M7HBCcnZIpTczuxC4EKBr1641EV/aufrqq1O2pkxEpEZUtb5o+Sk+yq8v2rdv2WbR/ffX+qKSVMlIzpYBXaK2c4Hl5U8ys77AI8Dx7r6mspu5+wRCnzTy8vJqt41WRERSV/T6oiefXLq/uvVFs7LC6NA+fcLkuSX32HPP8FXNo5JgyUjOpgE9zKw78C1wJnBW9Alm1hX4J3COuy+o/RBFRCRtVbW+aHQt2wcfwN//HgYmRGvVqmLCFv2+fXsNSJDdUuvJmbsXmtnlwKuEqTQedfe5ZnZx5PiDwM1Aa+B+C30BCmN1mBMREakR0euL/td/le4vKgqrH3z9NSxdGl4l77/6CqZMgQ0byt6rYcPQly06YYtO4rp2Dc2wIpVIyjxn7v4S8FK5fQ9Gvf858PPajktERKSMrKwwXUduLgwdGvuc9etjJ29Ll8Lrr8Py5aH/W7R27SpP3vbcM6xrqoEKGSuzVghIguHDh/PLX/6SY489dse+u+++mwULFnD//fdXes348ePJy8vjhBNO4JlnnqFly5Zlzhk3bhw5OTlce+21lX72pEmT6NmzJ7179wbg5ptvZtiwYRx11FG7VaboRdlFRDJeixal643GUlAAy5bFTt7mzoWXXoItW8pe07hx1clb586hhk7SkpKzBBs1ahQTJ04sk5yVTNwaj5deeqn6kyoxadIkTjrppB3J2W233bbL9xIRkV3UoEEYWBBZ47gCd1izpmLyVvJ19uwwt1u0kvVMq+r7Vu6Peqk7lJwl2IgRI/j1r3/Ntm3baNSoEUuWLGH58uUccsghXHLJJUybNo0tW7YwYsQIbr311grXd+vWjenTp9OmTRvuuOMOnnzySbp06ULbtm054IADgDCP2YQJE9i+fTv77LMPTz31FLNmzeKFF17g7bff5je/+Q3PPfcct99+OyeddBIjRozgjTfe4Nprr6WwsJADDzyQBx54gEaNGtGtWzfOO+88/v3vf1NQUMDf//53evXqVWn51q5dy5gxY1i8eDFNmjRhwoQJ9O3bl7fffpurrroKCOtyTp06lY0bNzJy5Eg2bNhAYWEhDzzwAIceemhivvEiInWFWVhPtE2bslOBRNuypbT2rXzyNmMGPP88bN9e9prmzSsmbNFJXKdOodlWUk5GJWdjx4aBODWpf/+q11Nv3bo1gwYN4pVXXuGUU05h4sSJjBw5EjPjjjvuoFWrVhQVFXHkkUcyZ84c+lZSLT5jxgwmTpzIJ598QmFhIQMHDtyRnJ1++ulccMEFAPz617/mz3/+M1dccQUnn3zyjmQs2tatWxk9ejRvvPEGPXv25Nxzz+WBBx5g7NixALRp04aZM2dy//33M378eB555JFKy3fLLbcwYMAAJk2axJtvvsm5557LrFmzGD9+PPfddx9Dhw5l48aNZGdnM2HCBI499lh+9atfUVRUxOaSYesiIlK1xo2hR4/wiqW4GL77LnbytnQpfPghrF1b9pqS/nRVDVzQtCFJkVHJWbKUNG2WJGePPvooAM8++ywTJkygsLCQFStW8Pnnn1eanL3zzjucdtppNGnSBICTo+bs+eyzz/j1r3/NunXr2LhxY5km1Fi++OILunfvTs+ePQE477zzuO+++3YkZ6effjoABxxwAP/85z+rvNe7777Lc5EJHY844gjWrFnD+vXrGTp0KNdccw1nn302p59+Orm5uRx44IGMGTOGgoICTj31VPr371/1N05EROJTrx506BBela2hvHFjmJQ3VvL2zjuhZq5kCawSrVrFbjIted+unaYNSYCMSs6qquFKpFNPPZVrrrmGmTNnsmXLFgYOHMhXX33F+PHjmTZtGnvssQejR49m69atVd7HKhm5M3r0aCZNmkS/fv14/PHHmTJlSpX3qW491UaRmbGzsrIoLD+/Txz3MjNuuOEGTjzxRF566SUOOuggJk+ezLBhw5g6dSr/+c9/OOecc7juuus499xzq7y/iIjUkJycMLluZesXFxWFkaWxBi4sWgRvvhkm8I3WsGHZmrbyyVuXLpo2ZBdkVHKWLDk5OQwfPpwxY8YwatQoADZs2EDTpk1p0aIFq1at4uWXX2b48OGV3mPYsGGMHj2aG264gcLCQv7973/vWCMzPz+fjh07UlBQwNNPP03nzmGN+GbNmpFf/gcJ6NWrF0uWLGHhwoU7+qgddthhu1S2YcOG8fTTT3PTTTcxZcoU2rRpQ/PmzVm0aBF9+vShT58+fPDBB8yfP5/GjRvTuXNnLrjgAjZt2sTMmTOVnImIpIqsrJBMdelS+bQh69ZVPnChsmlD2revuu+bpg2pQMlZLRk1ahSnn346EydOBKBfv34MGDCA/fffn7322ouhlf0gRAwcOJCRI0fSv39/9txzzzId6W+//XYGDx7MnnvuSZ8+fXYkZGeeeSYXXHAB99xzD//4xz92nJ+dnc1jjz3GGWecsWNAwMUXX7xL5Ro3bhznn38+ffv2pUmTJjzxxBNAmC7krbfeIisri969e3P88cfvGKXaoEEDcnJyePLJJ3fpM0VEJElatgyvyqYN2b4dvv22Ys3b119XPm1IkyZVJ2+5uWHEawax6pq46pK8vDyfPn16mX3z5s1jv8qqcCXl6HmJiKSx6GlDYvV9+/rrMLAhWsm0IVX1fWvRIjnl2U1mNiPWCkiqORMREZHasTPThsRK3qZNg3/+M/a0IbGSt5KvHTvWqWlDlJyJiIhI6oh32pDKat4++KDitCH165dOG1JZ82nTpokvW5yUnImIiEjdET1tyODBsc/Jzw/ThsRK3t55B/7614rThrRuXTZ5u+uupE0TkhHJmbtXOg2FpI506v8oIiJJ1KwZ9O4dXrEUFsKKFbGTt0WLwpJZf/xj7cYcJe2Ts+zsbNasWUPr1q2VoKUwd2fNmjVkaz4cERFJtPr1q582JInSPjnLzc1l2bJlrF69OtmhSDWys7PJzc1NdhgiIiJJlfbJWYMGDejevXuywxARERGJixbEEhEREUkhSs5EREREUoiSMxEREZEUklbLN5nZauDrBH9MG+D7BH9GqsrkskNmlz+Tyw6ZXX6VPXNlcvlrq+x7unvb8jvTKjmrDWY2PdY6WJkgk8sOmV3+TC47ZHb5VfbMLDtkdvmTXXY1a4qIiIikECVnIiIiIilEydnOm5DsAJIok8sOmV3+TC47ZHb5VfbMlcnlT2rZ1edMREREJIWo5kxEREQkhSg5i8HMHjWz78zss0qOm5ndY2YLzWyOmQ2s7RgTJY6yDzez9WY2K/K6ubZjTCQz62Jmb5nZPDOba2ZXxTgnLZ9/nGVPy+dvZtlm9rGZzY6U/dYY56Tlc4e4y5+Wz76EmWWZ2Sdm9mKMY2n77KHasqf7c19iZp9GyjY9xvGkPPu0X1tzFz0O3As8Wcnx44Eekddg4IHI13TwOFWXHeAddz+pdsKpdYXAf7v7TDNrBswws9fd/fOoc9L1+cdTdkjP578NOMLdN5pZA+BdM3vZ3T+MOiddnzvEV35Iz2df4ipgHtA8xrF0fvZQddkhvZ87wOHuXtmcZkl59qo5i8HdpwJrqzjlFOBJDz4EWppZx9qJLrHiKHtac/cV7j4z8j6f8B9W53KnpeXzj7PsaSnyLDdGNhtEXuU75Kblc4e4y5+2zCwXOBF4pJJT0vbZx1H2TJeUZ6/kbNd0Br6J2l5GhvwSizg40vzxspntn+xgEsXMugEDgI/KHUr7519F2SFNn3+kaWcW8B3wurtn1HOPo/yQps8euBu4Hiiu5Hg6P/u7qbrskL7PHcIfIa+Z2QwzuzDG8aQ8eyVnu8Zi7MuUvzJnEpab6Af8CZiU3HASw8xygOeAse6+ofzhGJekzfOvpuxp+/zdvcjd+wO5wCAz+1G5U9L6ucdR/rR89mZ2EvCdu8+o6rQY++r8s4+z7Gn53KMMdfeBhObLy8xsWLnjSXn2Ss52zTKgS9R2LrA8SbHUKnffUNL84e4vAQ3MrE2Sw6pRkT43zwFPu/s/Y5ySts+/urJnwvN393XAFOC4cofS9rlHq6z8afzshwInm9kSYCJwhJn9pdw56frsqy17Gj93ANx9eeTrd8DzwKBypyTl2Ss52zUvAOdGRnEcBKx39xXJDqo2mFkHM7PI+0GEf0NrkhtVzYmU7c/APHf/QyWnpeXzj6fs6fr8zaytmbWMvG8MHAXML3daWj53iK/86frs3f2X7p7r7t2AM4E33f2n5U5Ly2cfT9nT9bkDmFnTyOAnzKwpcAxQfqaCpDx7jdaMwcz+CgwH2pjZMuAWQgdZ3P1B4CXgBGAhsBk4PzmR1rw4yj4CuMTMCoEtwJmeXjMZDwXOAT6N9L8BuBHoCnX7+ZvZy8BEd3+iklOiy74B+AG4hLJlT6nnH/mL/+fuPnk3b9UReMLMsgi/fPYHzjezi4GzgNeB3xDjuZuZAz3cfeEuxH82cJ67H7Ob8e+u8uV/1t1fjJQ/JZ99opUre538md9VGfTc2wPPR3LP+sAz7v5KKjx7rRAgksLMbGPUZhPClAdFke2L3P3p2o8qddRgclb+vnEnXPGeGxlk8RXQwN0LayRQEUlLqjkTSWHunlPyvqpExMzq6xe+pAr9exTZPepzJlIHWZi1e5mZ/cLMVgKPmdkeZvaima02sx8i73OjrpliZj+PvB9tZu+a2fjIuV+Z2fG7eG53M5tqZvlmNtnM7ovRobrk3HhivN3M3ovc7zWL6nxsZueY2ddmtsbMflXF9+cgM1sZaaYr2Xeamc2JvB9kZh+Y2TozW2Fm95pZw0ru9biZ/SZq+7rINcvNbEy5c0+0MNP6BjP7xszGRR2eGvm6zsw2mtnBJd/bqOuHmNk0CzOyTzOzIfF+b3by+9zKzB6LlOEHM5sUdewUC7OlbzCzRWZ2XGT/EjM7Kuq8cSXP2cy6mZmb2c/MbCnwZmT/3yPPYX3k38j+Udc3NrP/izzP9ZF/Y43N7D9mdkW58swxs1NjlVUkHSk5E6m7OgCtgD2BCwk/z49FtrsS+ofcW8X1g4EvgDbA74A/m1msYePVnfsM8DHQGhhH6LdWmXhiPIvQr6Md0BC4FsDMehNm5z4H6BT5vFxiiEwWuQk4otx9n4m8LwKujpTnYOBI4NIq4iYSw3GReI4mzBh+VLlTNgHnAi0JE3teEpVUlAzRb+nuOe7+Qbl7twL+A9wTKdsfgP+YWetyZajwvYmhuu/zU4Rm8v0j97orEsMgwuog10XKMAxYUslnxHIYsB9wbGT7ZcL3qR1hSoboZvjxwAHAEMK/45K5tp4AdnRKN7N+hHmlXtqJOETqNCVnInVXMXCLu29z9y3uvsbdn3P3zZEZ/u8g/LKszNfu/rC7FxF+IXYkdJCN+1wz6wocCNzs7tvd/V3C6KaY4ozxMXdf4O5bgGeB/pH9I4AX3X2qu28DbqLqiTP/CowCsDAi64TIPtx9hrt/6O6F7r4EeChGHLH8VyS+z9x9EyEZjS7fFHf/1N2L3X1O5PPiuS+EZO5Ld38qEtdfCSMmfxx1TmXfmzKq+j5bmN38eOBid//B3Qvc/e3IpT8DHnX31yNl+Nbdy49arco4d98UiQ93f9Td8yPPaxzQz8xamFk9YAxwVeQzitz9/ch5/wJ6mFmPyD3PAf7m7tt3Ig6ROk3JmUjdtdrdt5ZsmFkTM3so0ky0gdCM1jK6aa+clSVv3H1z5G3OTp7bCVgbtQ/KzqZdRpwxrox6vzkqpk7R944kR1UN6X8GON3MGgGnAzPd/etIHD0jTX0rI3H8llCLVp0yMQBflyvfYAuLx682s/XAxXHet+TeX5fb9zVlZyOv7HtTRjXf5y6EZ/ZDjEu7AIvijDeWHd8bCysO3BlpGt1AaQ1cm8grO9ZnRRK0Z4GfRpK4UYSaPpGMoeRMpO4qP9T6v4F9gcHu3pzSZrTKmiprwgqglZk1idrXpbKT2b0YV0TfO/KZrSs72cOC7V8TaomimzQhNI/OJ4yybE6YLmWnYyAyzUiUZwg1h13cvQXwYNR9qxsav5zQDBmtK/BtHHGVV9X3+RvCM2sZ47pvgL0ruecmQlNoiQ4xzoku41mEdQmPAloA3aJi+B7YWsVnPQGcTWhu3ly+CVgk3Sk5E0kfzQh9i9ZF+i/dkugPjNRETQfGmVlDMzuYss1wNRnjP4CTzOyQSOf926j+/7BngCsJycnfy8WxAdhoZr0I87nF41lgtJn1jiSH5eNvRqiV2hrpv3VW1LHVhGbYvSq590tATzM7y8zqm9lIoDfwYpyxlY8j5vc5MoHmy8D9kYEDDax0yZo/E+Z3O9LM6plZ58j3B2AWcGbk/DxCM3N1MWwj1G42IdROlsRQDDwK/MHMOkVq2Q6O1HISScaKgf9DtWaSgZSciaSPu4HGhFqJD4FXaulzzyZ0ql9DmKj1b4RfyrHczS7G6O5zgcsICdcKwiS5y6q5rGRS5Tfd/fuo/dcSEqd84OFIzPHE8HKkDG8SJqV8s9wplwK3mVk+cDMhmSu5djOh79d7FkaJHlTu3muAkwi1XmsIHeRPKhd3vO6m6u/zOUABofbwO2BsJIaPCQMO7gLWA29TWpt3E6Gm6wfgVsrWRMbyJKHm8lvg80gc0a4FPgWmAWuB/6Xs76QngT5AzJG/IulMk9CKSI0ys78B89094TV3kr7M7FzgQnc/JNmxiNQ21ZyJyG4xswPNbO9IM9hxhH5Gk5IcltRhkSbjS4EJyY5FJBmUnInI7uoATAE2EubousTdP0lqRFJnmdmxhP55q6i+6VQkLalZU0RERCSFqOZMREREJIUoORMRERFJIfWTHUBNatOmjXfr1i3ZYYiIiIhUa8aMGd+7e9vy+9MqOevWrRvTp09PdhgiIiIi1TKz8ku2AWrWFBEREUkpSs5EREREUoiSMxEREZEUklZ9zkRERER2yvbtsGoVrFhR+tq4Ea69NmkhKTkTERGR9JOfX5psrVxZNvmK3l6zpuK1DRvCNddAveQ0MCo5ExERkbqhuBi+/77qZKtke9Omitc3bAgdOkDHjrDPPnDooaXbHTuWvm/XLmmJGSg5ExERkWTbtq1i02KshGvVKigsrHh98+alCdaBB5a+j064OnaEPfYAs9ov305SciYiIiI1zz3+psW1aytebwZt25YmVn37lk20ohOvJk1qv3wJpORMRERE4ldcDKtXx9e0uHlzxesbNixNrnr0gGHDYtdytWsH9TMzTcnMUouIiEhZ27aVTbAqS75WrYKioorXt2hRmlwNHlwx2SrZriNNi8mk5ExERCRducOGDfE1Lf7wQ8XrzUINVnTTYqxmxQ4d0q5pMZmUnImIiNQ1RUXxNy1u2VLx+kaNSpOqffeF4cNj13S1bZuxTYvJpO+4iIhIqti6Nb6mxe++q7xpsSSxOvjgyjvQt2yppsUUpuRMREQkkdxh/frqmxVXrIB16ypeX69eadNihw7Qv3/sDvQdOkDjxrVdOkkAJWciIiK7oqgo1GBVlXCVvN+6teL1JU2LHTvCfvvBEUfErulq1w6ysmq/fJI0Ss5ERESiuYeka/FiWL686qbF4uKK17dsWbZpsbIJUVu0UNOixKTkTEREMtO6dbBgAXz5ZcWvGzaUPbdePWjfvjS5Gjgwdgf69u3VtCi7TcmZiIikr02bYOHC2AnY6tWl55nBnnuGSVHPOSd83Xtv6Ny5dNSimhallig5ExGRum379tAEWT4BW7AAvv227LkdO0LPnnDKKeFrjx7h6157QXZ2cuIXKUfJmYiIpL6iIli6NHYz5JIlZft+tWoVEq4jjiibgO2zDzRrlrQiiMRLyZmIiKQG99DRPlYCtmhRqCEr0bRpSLgOPBDOOqs0CevRA1q3Tl4ZRGqAkjMREalda9ZUbH788svw2rSp9LxGjUK/r333hZNOKlsL1qGDRjpK2lJyJiIiNS8/P3Yn/AULyq7hmJUF3bqFhOuww0qTrx49oEsXdcKXjKTkTEREds3WraG5MVYt2MqVZc/t0iUkXCNHlk3AuneHhg2TE79IilJyJiIilSsoCB3uY9WCLV0a+omVaNcuJFzHH1+agPXsGZommzRJWhFE6holZyIima64GJYti52ALV4MhYWl57ZoERKuoUPh/PPL1oK1aJG8MoikESVnIiKZoGRJovLNjwsWhElao9d+bNw4JFt9+sBPflK2FqxNG3XEF0kwJWciIukk3iWJGjQIE6/27AnHHFN2JGSnTmG5IhFJioQmZ2Z2HPBHIAt4xN3vLHd8D+BRYG9gKzDG3T+L51oRkYxV2ZJECxbA99+XnleyJFHPnnDQQaW1Xz16hP319fe5SCpK2E+mmWUB9wFHA8uAaWb2grt/HnXajcAsdz/NzHpFzj8yzmtFRNJXyZJEsWrByi9J1KlTSLhOO61sHzAtSSRSJyXyz6ZBwEJ3XwxgZhOBU4DoBKs38D8A7j7fzLqZWXtgrziuFRGp23ZmSaLWrUPCdeSRZRMwLUkkknYSmZx1Br6J2l4GDC53zmzgdOBdMxsE7AnkxnmtiEjqi7UkUcn78ksS5eSULkl09tllk7BWrZJXBhGpVYlMzmIN5/Fy23cCfzSzWcCnwCdAYZzXhg8xuxC4EKBr1667GquIyK5zD0sSxeqEH2tJon32CUsS/fjHZRMwLUkkIiQ2OVsGdInazgWWR5/g7huA8wHMzICvIq8m1V0bdY8JwASAvLy8mAmciEiNiLUkUcn78ksSde9euiRR9EjI3FwtSSQiVUpkcjYN6GFm3YFvgTOBs6JPMLOWwGZ33w78HJjq7hvMrNprRUQSpqgI5s2DDz6AadNg/vzKlyTq2TMsSRSdgHXvHqaqEBHZBQlLzty90MwuB14lTIfxqLvPNbOLI8cfBPYDnjSzIkJn/59VdW2iYhWRDLd2LXz0UUjGPvgAPv64dE6wVq2gd++wJFH0VBRakkhEEsTc06clMC8vz6dPn57sMEQklRUVwdy5IQn78MPw9YsvwrF69aBv3zAn2MEHh9c++6gfmIgkhJnNcPe88vs1A6GIpLfvvw9JWEki9vHHsHFjONamTUjAzjsvJGQHHhhGTIqIJJGSMxFJH4WF8Nlnpc2TH34Y+opB6ITfrx+ce25prdhee6lWTERSjpIzEam7Vq8u2zw5bVrptBXt2oUE7Gc/C7VieXnQtGly4xURiYOSMxGpGwoK4NNPy9aKLVoUjtWvD/37w/nnl9aKdeumWjERqZOUnIlIalq1qmKt2JYt4ViHDiEBu+iiUCt2wAEaOSkiaUPJmYgkX0EBzJ5dtlbsq6/CsQYNYMAAuOCC0lqxrl1VKyYiaUvJmYjUvhUrytaKTZ8OW7eGY506hQTssstCrdjAgdC4cXLjFRGpRUrORCSxtm+HWbNKa8U++ACWLg3HGjYMydfFF5fWiuXmqlZMRDKakjMRqVnfflu2VmzGDNi2LRzLzQ0J2NixoVZswADIzk5quCIiqUbJmYjsum3bYObM0kTsgw9g2bJwrFGj0FH/sstCQnbQQSE5ExGRKik5E5H4ffNN2U77M2eGZksInfSHDi1NxPr3DwmaiIjsFCVnIhLb1q2hSbKkVuzDD0OTJYSmyLw8uPLK0mSsU6fkxisikiaUnIkIuIdO+tG1Yp98Eqa4gDCh67BhpYlYv36hM7+IiNQ4JWcimWjLljB9RXRfsZUrw7HGjcMC4FdfXZqMdeiQ3HhFRDJItcmZmZ0EvOTuxbUQj4jUNHdYsqRsrdisWWGRcAiLfx95ZGki1rdvmPhVRESSIp6aszOBP5rZc8Bj7j4vwTGJyO7YtKlsrdiHH4alkCAscTRoEFx7bWky1q5dcuMVEZEyqk3O3P2nZtYcGAU8ZmYOPAb81d3zEx2giFTBHRYvLlsrNns2FBWF4/vsA8ccU5qI9ekTFgkXEZGUFdf/0u6+IVJz1hgYC5wGXGdm97j7nxIYn4hE27gxLAAeXSu2enU4lpMTasV+8YvSZKxNm+TGKyIiOy2ePmc/BsYAewNPAYPc/TszawLMA5SciSSCOyxcWLZWbM4cKI50/+zZE044oTQR+9GPICsruTGLiMhui6fm7AzgLnefGr3T3Teb2ZjEhCWSgfLz4eOPy9aKrVkTjjVrBoMHw403hmRs8GBo3Tq58YqISELEk5zdAqwo2TCzxkB7d1/i7m8kLDKRdFZcDAsWlE3EPvustFasVy84+eTSWrHevVUrJiKSIeJJzv4ODInaLorsOzAhEYmkow0b4KOPShOxDz+EH34Ix1q0CDVhp55aWiu2xx5JDVdERJInnuSsvrtvL9lw9+1mpqnBRSpTXAzz55etFZs7N/Qhg1ALdvrppbVi++0H9eolN2YREUkZ8SRnq83sZHd/AcDMTgG+T2xYInXIunVla8U++ijsA2jZMiRgI0aEZGzQoLBPRESkEvEkZxcDT5vZvYAB3wDnJjQqkVRVXAyff1522aN5kXmZzWD//eGMM0prxfbdV7ViIiKyU+KZhHYRcJCZ5QCmiWcl42zeDPfeC5Mnh1qxDRvC/latQgI2alRprVjz5smNVURE6ry4JqE1sxOB/YFsMwPA3W+L47rjgD8CWcAj7n5nueMtgL8AXSOxjHf3xyLHlgD5hAEIhe6eF1+RRGqIOzz/fFgAfOnSsOZkSSJ20EFhnrHIz4OIiEhNiWcS2geBJsDhwCPACODjOK7LAu4DjgaWAdPM7AV3/zzqtMuAz939x2bWFvjCzJ6OGoBwuLurf5vUvgUL4Ior4LXXwpJHb78Nw4YlOyoR2U0FBWH52c2bK3+VHN+6NfRkcC8dz1PyPvpV2f6aviYVYsiUaxo3Dj1YkiWemrMh7t7XzOa4+61m9n/AP+O4bhCw0N0XA5jZROAUILq4DjSzUB2XA6wFCneqBCI1adMmuOMOGD8+/HTefTdcdpnWoxRJMPeQDMWTNO3OOYW1/BumpHLdrOKrsv119ZrdvV+9eqlT1oZJnpMint84WyNfN5tZJ2AN0D2O6zoTBg+UWAYMLnfOvcALwHKgGTDS3SOzcOLAa5GF1h9y9wlxfKbIrnGH556Da66Bb76Bc8+F//1f6NCBggLIXxtOa9gQGjQIL/Xzl0xRVARbtuxeUhRPYlVSkxEvM2jSJPZrjz2gc+fwvmnTys+r6nh2dvwJQ6z9IrsqnuTs32bWEvg9MJOQND0cx3Wx/mmW/9E7FpgFHEFYu/N1M3vH3TcAQ919uZm1i+yfX34JKQAzuxC4EKBr165xhCWZwD38MtiwoewrPz/G9tc/sGHKDDasbMiGps+R33M/Nryfw4Z+4ZytW2N/RlZWaaIWnbTVlfdKLtNDQUHia5sq+xmoSr16IemJlfi0b199YhTP8exsJUGSnqpMzsysHvCGu68DnjOzF4Fsd18fx72XAV2itnMJNWTRzgfudHcHFprZV0Av4GN3Xw4QWWT9eUIzaYXkLFKjNgEgLy9vJ//uklSzbVvFBKrSpKqKc/Lz4/srvFFWAc2LCmhWb2+ad86h+V5t6NTC6NU8LGfZvHl4NWsWzi8oCK/t2yu+j7Uv+v3WrSG2qs6Jfp9o1SWXqZBAVva+LiSX7uHfc6Jrm3bl30rDhpUnP61bx58cVXW8QQMlTiK7qsrkzN2LI33MDo5sbwO2xXnvaUAPM+sOfAucCZxV7pylwJHAO2bWHtgXWGxmTYF67p4feX8MUO3oUEmOoiLYuLFmkqrt26v/PLPSpKkkcWrRArp0Kd2OPl5hXzOn2Zv/otkt19Bo+VcwejTceSe0b5vw71W83MP3dVeSwJp6X37fli2lz6i6e9RmcpnoRLB+/dhJVjyJU8lSqTujcePYyU+LFtCx487VLMU63rixulCKpLp4fkRfM7OfAP+M1HDFxd0Lzexy4FXCVBqPuvtcM7s4cvxB4HbgcTP7lNAM+gt3/97M9gKej0zbUR94xt1f2amSSZVKOt/GkzBVd87GjfF9ZuPGFROmPfesJIGqZF+zZuGXzS7/RT5vHlxwObz5JvTvD39/D4YMqfay2mYWfoHW1V+i7qHj9a4kgYl6X1LLFM+5sTqNm1VMdEq227bdtWa58sdL+jiJSGaz6vItM8sHmhJGUW4lJFHu7ik322ZeXp5Pnz492WEkVGFhaVK0u0lVPKOWsrLiS5jiOSepiUZ+Ptx2Wxh9mZMTRmRedFEooEg57qUJW2FhSJoaNlQznYjULDObEWse13hWCGiWmJAyh3v4qz3epr2qztm8Ob7PbNq0YsLUtu3OJ1V1vsOtO0ycCNdeC8uXw5gxoQmzbeo0YUrqKRlKn+zh9CKSmeKZhDbmzJuxRk6mu+nT4bvvdi2piqfvSYMGoV9JdILUoQP06LFztVc5OaoQAmDuXLj8cpgyBQYODFNlHHRQsqMSERGpUjwNTddFvc8mjJqcQZj+IqP8/Ocwe3bZfWaliVF0gtSpU/zNfSXvGzVKTrnSzoYNcOut8Mc/hm/sAw/ABRcoYxURkTohnmbNH0dvm1kX4HcJiyiFPfJIqAGLTqqaNlUH3pThDs88E5owV60K2fRvfwtt2iQ7MhERkbjtShftZcCPajqQuiBPS6+nrk8/DU2YU6eGB/Wvf8GgQcmOSkREZKfF0+fsT5TO7F8P6A/MrvQCkdq0fj3ccgvce2/osPfQQ/Czn6kJU0RE6qx4as6i56YoBP7q7u8lKB6R+LjDU0/B9deHURoXXhimx2jdOtmRiYiI7JZ4krN/AFvdvQjAzLLMrIm7xzmpg0gNmz0bLrsM3nsvNF2++KLanEVEJG3E05X9DaBx1HZjYHJiwhGpwrp1cOWVYVqML74IIzQ++ECJmYiIpJV4as6y3X3HAj3uvtHMmiQwJpGyiovhySfhF7+A1avhkkvg9tuhVatkRyYiIlLj4qk522RmA0s2zOwAYEviQhKJMnMmHHIInH8+7LVXmAn4vvuUmImISNqKp+ZsLPB3M1se2e4IjExYRCIAP/wAv/41PPhg6OT/6KNw3nmaVE5ERNJePJPQTjOzXsC+hEXP57t7QcIjk8xUXAyPPQY33ABr18Kll4YFy/fYI9mRiYiI1IpqqyHM7DKgqbt/5u6fAjlmdmniQ5OMM2MGDBkSZvbfd9+w/ac/KTETEZGMEk8b0QXuvq5kw91/AC5IWESSedasgYsvhgMPhK++gieegHfegf79kx2ZiIhIrYsnOatnZlayYWZZQMPEhSQZo7gYHn441JI9/DBccUWYIuPcc8OK8iIiIhkongEBrwLPmtmDhGWcLgZeTmhUkv6mTQsTyU6bFkZj3ncf9O2b7KhERESSLp6as18QJqK9BLgMmEPZSWlF4vf992GppcGD4ZtvwhJMU6cqMRMREYmoNjlz92LgQ2AxkAccCcxLcFySboqKwrQY++4bpsUYOzY0Yf70p2rCFBERiVJps6aZ9QTOBEYBa4C/Abj74bUTmqSNjz4KTZgzZsBhh8G998KPfpTsqERERFJSVTVn8wm1ZD9290Pc/U9AUe2EJWlh9eowLcZBB8Hy5fDMM/DWW0rMREREqlBVcvYTYCXwlpk9bGZHEiahFalaURHcfz/07Bmmxbj22tCEOWqUmjBFRESqUWly5u7Pu/tIoBcwBbgaaG9mD5jZMbUUn9Q1H3wQ5iu77DIYMABmz4bf/x6aNUt2ZCIiInVCPAMCNrn70+5+EpALzAJuSHRgUsd8911YnHzIkPB+4kR44w3o3TvZkYmIiNQpO7WKtLuvdfeH3P2IRAUkdUxhYVhiqWdP+Mtf4PrrYf58GDlSTZgiIiK7IJ5JaEVie++90Hw5ezYcdVRI0nr1SnZUIiIiddpO1ZztLDM7zsy+MLOFZlahKdTMWpjZv81stpnNNbPz471WkmjVKjjvvDCz/5o18Pe/w2uvKTETERGpAQlLziJrcN4HHA/0BkaZWfkOSJcBn7t7P2A48H9m1jDOa6W2FRbCH/8YmjD/+le44YbQhDlihJowRUREakgia84GAQvdfbG7bwcmAqeUO8eBZpGF1XOAtUBhnNdKbZo6FQYODDP7H3QQfPop/M//QNOmyY5MREQkrSQyOesMfBO1vSyyL9q9wH7AcuBT4KrIclHxXAuAmV1oZtPNbPrq1atrKnYpsWJFWGLpsMNg/Xp47jl45ZWwDJOIiIjUuEQmZ7Haubzc9rGEqTk6Af2Be82seZzXhp3uE9w9z93z2rZtu+vRSlkFBXDXXSEJ+/vf4Ve/gnnz4PTT1YQpIiKSQIkcrbkM6BK1nUuoIYt2PnCnuzuw0My+Ikx6G8+1kihvvx1GYc6dC8cfH/qZ9eiR7KhEREQyQiJrzqYBPcysu5k1JCyi/kK5c5YS1u/EzNoD+wKL47xWatry5XDWWTB8OGzcCJMmwX/+o8RMRESkFiWs5szdC83scuBVIAt41N3nmtnFkeMPArcDj5vZp4SmzF+4+/cAsa5NVKwZr6Ag1I7demt4f9NNYSRmkybJjkxERCTjWGhRTA95eXk+ffr0ZIdRt7z5Jlx+eehPduKJIUnbe+9kRyUiIpL2zGyGu+eV35/QSWglhS1bBmeeCUceCVu3wgsvwIsvKjETERFJMi3flGm2b4e774bbboOiIhg3LqyH2bhxsiMTEZFdUFBQwLJly9i6dWuyQ5FKZGdnk5ubS4MGDeI6X8lZJpk8Ga64Iszq/+MfhyRtr72SHZWIiOyGZcuW0axZM7p164ZpqqOU4+6sWbOGZcuW0b1797iuUbNmJvjmGzjjDDj66FBz9uKLoRlTiZmISJ23detWWrdurcQsRZkZrVu33qmaTSVn6WzbtrDEUq9eISG77bYwd9mJJyY7MhERqUFKzFLbzj4fJWfp6rXXoG9fuPFGOOaYMBrzppsgOzvZkYmISBpZs2YN/fv3p3///nTo0IHOnTvv2N6+fXuV106fPp0rr7yy2s8YMmRITYVbJ6jPWbpZuhSuvhr++U/YZx94+WU47rhkRyUiImmqdevWzJo1C4Bx48aRk5PDtddeu+N4YWEh9evHTjfy8vLIy6swk0QF77//fo3EWleo5ixdbNsGd9wRmjBffhl+8xv47DMlZiIiUutGjx7NNddcw+GHH84vfvELPv74Y4YMGcKAAQMYMmQIX3zxBQBTpkzhpJNOAkJiN2bMGIYPH85ee+3FPffcs+N+OTk5O84fPnw4I0aMoFevXpx99tmUzNf60ksv0atXLw455BCuvPLKHfeNtmTJEg499FAGDhzIwIEDyyR9v/vd7+jTpw/9+vXjhhtuAGDhwoUcddRR9OvXj4EDB7Jo0aLEfMPKUc1ZOnj5ZbjySli4MCxM/oc/wJ57JjsqERGpbWPHQqQWq8b07x9G9++kBQsWMHnyZLKystiwYQNTp06lfv36TJ48mRtvvJHnnnuuwjXz58/nrbfeIj8/n3333ZdLLrmkwvQTn3zyCXPnzqVTp04MHTqU9957j7y8PC666CKmTp1K9+7dGTVqVMyY2rVrx+uvv052djZffvklo0aNYvr06bz88stMmjSJjz76iCZNmrB27VoAzj77bG644QZOO+00tm7dSnFx8U5/H3aFkrO6bMmS8IP4r39Bz57w6quhf5mIiEiSnXHGGWRlZQGwfv16zjvvPL788kvMjIKCgpjXnHjiiTRq1IhGjRrRrl07Vq1aRW5ubplzBg0atGNf//79WbJkCTk5Oey11147pqoYNWoUEyZMqHD/goICLr/8cmbNmkVWVhYLFiwAYPLkyZx//vk0iSxb2KpVK/Lz8/n222857bTTgDBXWW1RclYXbd0Kv/89/Pa3UK9eGJF59dXQqFGyIxMRkWTahRquRGnatOmO9zfddBOHH344zz//PEuWLGH48OExr2kU9XssKyuLwsLCuM6JdynKu+66i/bt2zN79myKi4t3JFzuXmFEZTKXt1Sfs7rmP/+BH/0Ibr45TCQ7f35YpFyJmYiIpKj169fTuXNnAB5//PEav3+vXr1YvHgxS5YsAeBvf/tbpXF07NiRevXq8dRTT1FUVATAMcccw6OPPsrmzZsBWLt2Lc2bNyc3N5dJkyYBsG3bth3HE03JWV2xeDGcfDKcdBLUrw+vvw7PPgtduiQ7MhERkSpdf/31/PKXv2To0KE7EqKa1LhxY+6//36OO+44DjnkENq3b0+LFi0qnHfppZfyxBNPcNBBB7FgwYIdtXvHHXccJ598Mnl5efTv35/x48cD8NRTT3HPPffQt29fhgwZwsqVK2s89lgsmdV2NS0vL8+nT5+e7DBq1pYt8L//C3feGZKym28O/cwaNkx2ZCIikgLmzZvHfvvtl+wwkm7jxo3k5OTg7lx22WX06NGDq6++Otlh7RDrOZnZDHevMJeIas5S2b//DfvvD7feCqeeGpowr79eiZmIiEg5Dz/8MP3792f//fdn/fr1XHTRRckOaZdpQEAqWrQIrroq9C/bbz944w044ohkRyUiIpKyrr766pSqKdsdqjlLJZs3h2bL/feHt9+G8eNh9mwlZiIiIhlENWepwD3MVTZ2LHz9NZx1Vpgqo1OnZEcmIiIitUw1Z8n25Zdwwglw2mnQrBlMmQJPP63ETEREJEMpOUuWTZvgV78Kc5a99x7cdRfMnAmHHZbsyERERCSJlJzVNnd47rnQ0f+3v4X/+i/44ovQpFlu/TAREZFUN3z4cF599dUy++6++24uvfTSKq8pmfrqhBNOYN26dRXOGTdu3I75xiozadIkPv/88x3bN998M5MnT96J6FOTkrPa9MUXcOyxMGIEtGwJU6fCU09Bx47JjkxERGSXjBo1iokTJ5bZN3HixEoXHy/vpZdeomXLlrv02eWTs9tuu42jjjpql+6VSpSc1YaNG8MSS336wEcfwR//GJowDz002ZGJiIjslhEjRvDiiy+ybds2AJYsWcLy5cs55JBDuOSSS8jLy2P//ffnlltuiXl9t27d+P777wG444472HfffTnqqKP44osvdpzz8MMPc+CBB9KvXz9+8pOfsHnzZt5//31eeOEFrrvuOvr378+iRYsYPXo0//jHPwB44403GDBgAH369GHMmDE74uvWrRu33HILAwcOpE+fPsyfP79CTEuWLOHQQw9l4MCBDBw4kPfff3/Hsd/97nf06dOHfv36ccMNNwCwcOFCjjrqKPr168fAgQNZtGjRbn1PNVozkdzhH/+Aa66BZcvgvPPCbP/t2yc7MhERSUNjx8KsWTV7z/79q15PvXXr1gwaNIhXXnmFU045hYkTJzJy5EjMjDvuuINWrVpRVFTEkUceyZw5c+jbt2/M+8yYMYOJEyfyySefUFhYyMCBAznggAMAOP3007ngggsA+PWvf82f//xnrrjiCk4++WROOukkRowYUeZeW7duZfTo0bzxxhv07NmTc889lwceeICxY8cC0KZNG2bOnMn999/P+PHjeeSRR8pc365dO15//XWys7P58ssvGTVqFNOnT+fll19m0qRJfPTRRzRp0oS1a9cCcPbZZ3PDDTdw2mmnsXXrVoqLi3f+Gx1FNWeJMm8eHH106FPWpg28+y48/rgSMxERSTvRTZvRTZrPPvssAwcOZMCAAcydO7dME2R577zzDqeddhpNmjShefPmnHzyyTuOffbZZxx66KH06dOHp59+mrlz51YZzxdffEH37t3p2bMnAOeddx5Tp07dcfz0008H4IADDtixWHq0goICLrjgAvr06cMZZ5yxI+7Jkydz/vnn06RJEwBatWpFfn4+3377LaeddhoA2dnZO47vKtWc1bT8fLj99jD6smlT+NOf4OKLw7qYIiIiCVRVDVcinXrqqVxzzTXMnDmTLVu2MHDgQL766ivGjx/PtGnT2GOPPRg9ejRbt26t8j5mFnP/6NGjmTRpEv369ePxxx9nypQpVd6nunXDGzVqBEBWVhaFhYUVjt911120b9+e2bNnU1xcTHZ29o77lo8xEWuUJ7TmzMyOM7MvzGyhmd0Q4/h1ZjYr8vrMzIrMrFXk2BIz+zRyLPVXM3eHiROhV68wgew558CCBXD55UrMREQkreXk5DB8+HDGjBmzo9Zsw4YNNG3alBYtWrBq1SpefvnlKu8xbNgwnn/+ebZs2UJ+fj7//ve/dxzLz8+nY8eOFBQU8PTTT+/Y36xZM/Lz8yvcq1evXixZsoSFCxcC8NRTT3HYTkxVtX79ejp27Ei9evV46qmnKCoqAuCYY47h0UcfZfPmzQCsXbuW5s2bk5uby6RJkwDYtm3bjuO7KmHJmZllAfcBxwO9gVFm1jv6HHf/vbv3d/f+wC+Bt919bdQph0eOV1ixPaXMnQtHHgmjRoVmy/ffh0cfhXbtkh2ZiIhIrRg1ahSzZ8/mzDPPBKBfv34MGDCA/fffnzFjxjB06NAqrx84cCAjR46kf//+/OQnP+HQqEFzt99+O4MHD+boo4+mV69eO/afeeaZ/P73v2fAgAFlOuFnZ2fz2GOPccYZZ9CnTx/q1avHxRdfHHdZLr30Up544gkOOuggFixYQNOmTQE47rjjOPnkk8nLy6N///47pvp46qmnuOeee+jbty9Dhgxh5cqVcX9WLJaI6jgAMzsYGOfux0a2fwng7v9TyfnPAG+5+8OR7SVAnrt/H+9n5uXlecm8KbUiPx9uvTWMvmzWDO64Ay68ELKyai8GERHJaPPmzWO//fZLdhhSjVjPycxmxKqASmSzZmfgm6jtZZF9FZhZE+A44Lmo3Q68ZmYzzOzCyj7EzC40s+lmNn316tU1EHYc3OGZZ2DffeH//g9Gjw5zmF1yiRIzERER2S2JTM5i9eqrrJrux8B75Zo0h7r7QEKz6GVmNizWhe4+wd3z3D2vbdu2uxdxPD77DA4/HM4+O6x/+eGH8PDDUBufLSIiImkvkcnZMqBL1HYusLySc88E/hq9w92XR75+BzwPDEpAjPErKgrzlfXvD59+Cg8+GCaUHTw4qWGJiIhIeklkcjYN6GFm3c2sISEBe6H8SWbWAjgM+FfUvqZm1qzkPXAM8FkCY61eVhZ88w387GdhFOZFF6kJU0REUkKi+o9LzdjZ55OwOR7cvdDMLgdeBbKAR919rpldHDn+YOTU04DX3H1T1OXtgecjc4nUB55x91cSFWvcJk5UQiYiIiklOzubNWvW0Lp160rnCZPkcXfWrFmzY660eCRstGYy1PpoTRERkSQrKChg2bJl1U7wKsmTnZ1Nbm4uDRo0KLO/stGamh1VRESkDmvQoAHdu3dPdhhSg7S2poiIiEgKUXImIiIikkKUnImIiIikkLQaEGBmq4GvE/wxbYC4l5RKM5lcdsjs8mdy2SGzy6+yZ65MLn9tlX1Pd68wi31aJWe1wcymp/xC7AmSyWWHzC5/JpcdMrv8Kntmlh0yu/zJLruaNUVERERSiJIzERERkRSi5GznTUh2AEmUyWWHzC5/JpcdMrv8KnvmyuTyJ7Xs6nMmIiIikkJUcyYiIiKSQpScxWBmj5rZd2b2WSXHzczuMbOFZjbHzAbWdoyJEkfZh5vZejObFXndXNsxJpKZdTGzt8xsnpnNNbOrYpyTls8/zrKn5fM3s2wz+9jMZkfKfmuMc9LyuUPc5U/LZ1/CzLLM7BMzezHGsbR99lBt2dP9uS8xs08jZauwOHeynr3W1oztceBe4MlKjh8P9Ii8BgMPRL6mg8epuuwA77j7SbUTTq0rBP7b3WeaWTNghpm97u6fR52Trs8/nrJDej7/bcAR7r7RzBoA75rZy+7+YdQ56frcIb7yQ3o++xJXAfOA5jGOpfOzh6rLDun93AEOd/fK5jRLyrNXzVkM7j4VWFvFKacAT3rwIdDSzDrWTnSJFUfZ05q7r3D3mZH3+YT/sDqXOy0tn3+cZU9LkWe5MbLZIPIq3yE3LZ87xF3+tGVmucCJwCOVnJK2zz6Osme6pDx7JWe7pjPwTdT2MjLkl1jEwZHmj5fNbP9kB5MoZtYNGAB8VO5Q2j//KsoOafr8I007s4DvgNfdPaOeexzlhzR99sDdwPVAcSXH0/nZ303VZYf0fe4Q/gh5zcxmmNmFMY4n5dkrOds1FmNfpvyVOZOw3EQ/4E/ApOSGkxhmlgM8B4x19w3lD8e4JG2efzVlT9vn7+5F7t4fyAUGmdmPyp2S1s89jvKn5bM3s5OA79x9RlWnxdhX5599nGVPy+ceZai7DyQ0X15mZsPKHU/Ks1dytmuWAV2itnOB5UmKpVa5+4aS5g93fwloYGZtkhxWjYr0uXkOeNrd/xnjlLR9/tWVPROev7uvA6YAx5U7lLbPPVpl5U/jZz8UONnMlgATgSPM7C/lzknXZ19t2dP4uQPg7ssjX78DngcGlTslKc9eydmueQE4NzKK4yBgvbuvSHZQtcHMOpiZRd4PIvwbWpPcqGpOpGx/Bua5+x8qOS0tn388ZU/X529mbc2sZeR9Y+AoYH6509LyuUN85U/XZ+/uv3T3XHfvBpwJvOnuPy13Wlo++3jKnq7PHcDMmkYGP2FmTYFjgPIzFSTl2Wu0Zgxm9ldgONDGzJYBtxA6yOLuDwIvAScAC4HNwPnJibTmxVH2EcAlZlYIbAHO9PSayXgocA7waaT/DcCNQFdI++cfT9nT9fl3BJ4wsyzCL59n3f1FM7sY0v65Q3zlT9dnH1MGPfsKMui5tweej+Se9YFn3P2VVHj2WiFAREREJIWoWVNEREQkhSg5ExEREUkhSs5EREREUoiSMxEREZEUouRMREREJIUoORORtGZmRWY2K+p1Qw3eu5uZlZ8XSURkt2ieMxFJd1siyxKJiNQJqjkTkYxkZkvM7H/N7OPIa5/I/j3N7A0zmxP52jWyv72ZPR9ZAHq2mQ2J3CrLzB42s7lm9lpkhn3M7Eoz+zxyn4lJKqaI1EFKzkQk3TUu16w5MurYBncfBNwL3B3Zdy/wpLv3BZ4G7onsvwd4O7IA9EBgbmR/D+A+d98fWAf8JLL/BmBA5D4XJ6ZoIpKOtEKAiKQ1M9vo7jkx9i8BjnD3xZEF31e6e2sz+x7o6O4Fkf0r3L2Nma0Gct19W9Q9ugGvu3uPyPYvgAbu/hszewXYCEwCJpUsHi0iUh3VnIlIJvNK3ld2Tizbot4XUdqX90TgPuAAYIaZqY+viMRFyZmIZLKRUV8/iLx/Hzgz8v5s4N3I+zeASwDMLMvMmld2UzOrB3Rx97eA64GWQIXaOxGRWPSXnIiku8ZmNitq+xV3L5lOo5GZfUT4Q3VUZN+VwKNmdh2wGjg/sv8qYIKZ/YxQQ3YJsKKSz8wC/mJmLQAD7nL3dTVUHhFJc+pzJiIZKdLnLM/dv092LCIi0dSsKSIiIpJCVHMmIiIikkJUcyYiIiKSQpSciYiIiKQQJWciIiIiKUTJmYiIiEgKUXImIiIikkKUnImIiIikkP8HpPcT27S6g00AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "print(history_dict.keys())\n",
    "\n",
    "acc = history_dict['binary_accuracy']\n",
    "val_acc = history_dict['val_binary_accuracy']\n",
    "loss = history_dict['loss']\n",
    "val_loss = history_dict['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "fig = plt.figure(figsize=(10, 6))\n",
    "fig.tight_layout()\n",
    "\n",
    "plt.subplot(2, 1, 1)\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, loss, 'r', label='Training loss')\n",
    "# b is for \"solid blue line\"\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "# plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(2, 1, 2)\n",
    "plt.plot(epochs, acc, 'r', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(loc='lower right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WzJZCo-cf-Jf"
   },
   "source": [
    "In this plot, the red lines represents the training loss and accuracy, and the blue lines are the validation loss and accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rtn7jewb6dg4"
   },
   "source": [
    "## Export for inference\n",
    "\n",
    "Now you just save your fine-tuned model for later use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 7: Write code to save the model to saved_model_path**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:32:02.391842Z",
     "iopub.status.busy": "2021-03-04T02:32:02.391132Z",
     "iopub.status.idle": "2021-03-04T02:32:08.584141Z",
     "shell.execute_reply": "2021-03-04T02:32:08.583571Z"
    },
    "id": "ShcvqJAgVera"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:Found untraced functions such as restored_function_body, restored_function_body, restored_function_body, restored_function_body, restored_function_body while saving (showing 5 of 310). These functions will not be directly callable after loading.\n",
      "WARNING:absl:Found untraced functions such as restored_function_body, restored_function_body, restored_function_body, restored_function_body, restored_function_body while saving (showing 5 of 310). These functions will not be directly callable after loading.\n"
     ]
    }
   ],
   "source": [
    "dataset_name = 'imdb'\n",
    "saved_model_path = './{}_bert'.format(dataset_name.replace('/', '_'))\n",
    "\n",
    "classifier_model.save(saved_model_path, include_optimizer=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PbI25bS1vD7s"
   },
   "source": [
    "Let's reload the model so you can try it side by side with the model that is still in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:32:08.589521Z",
     "iopub.status.busy": "2021-03-04T02:32:08.588938Z",
     "iopub.status.idle": "2021-03-04T02:32:15.921178Z",
     "shell.execute_reply": "2021-03-04T02:32:15.920565Z"
    },
    "id": "gUEWVskZjEF0"
   },
   "outputs": [],
   "source": [
    "reloaded_model = tf.saved_model.load(saved_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oyTappHTvNCz"
   },
   "source": [
    "Here you can test your model on any sentence you want, just add to the examples variable below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:32:15.928344Z",
     "iopub.status.busy": "2021-03-04T02:32:15.927703Z",
     "iopub.status.idle": "2021-03-04T02:32:16.706968Z",
     "shell.execute_reply": "2021-03-04T02:32:16.706338Z"
    },
    "id": "VBWzH6exlCPS"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results from the saved model:\n",
      "input: this is such an amazing movie! : score: 0.999552\n",
      "input: The movie was great!           : score: 0.991094\n",
      "input: The movie was meh.             : score: 0.645051\n",
      "input: The movie was okish.           : score: 0.007991\n",
      "input: The movie was terrible...      : score: 0.000922\n",
      "\n",
      "Results from the model in memory:\n",
      "input: this is such an amazing movie! : score: 0.999552\n",
      "input: The movie was great!           : score: 0.991094\n",
      "input: The movie was meh.             : score: 0.645051\n",
      "input: The movie was okish.           : score: 0.007991\n",
      "input: The movie was terrible...      : score: 0.000922\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def print_my_examples(inputs, results):\n",
    "  result_for_printing = \\\n",
    "    [f'input: {inputs[i]:<30} : score: {results[i][0]:.6f}'\n",
    "                         for i in range(len(inputs))]\n",
    "  print(*result_for_printing, sep='\\n')\n",
    "  print()\n",
    "\n",
    "\n",
    "examples = [\n",
    "    'this is such an amazing movie!',  # this is the same sentence tried earlier\n",
    "    'The movie was great!',\n",
    "    'The movie was meh.',\n",
    "    'The movie was okish.',\n",
    "    'The movie was terrible...'\n",
    "]\n",
    "\n",
    "reloaded_results = tf.sigmoid(reloaded_model(tf.constant(examples)))\n",
    "original_results = tf.sigmoid(classifier_model(tf.constant(examples)))\n",
    "\n",
    "print('Results from the saved model:')\n",
    "print_my_examples(examples, reloaded_results)\n",
    "print('Results from the model in memory:')\n",
    "print_my_examples(examples, original_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3cOmih754Y_M"
   },
   "source": [
    "If you want to use your model on [TF Serving](https://www.tensorflow.org/tfx/guide/serving), remember that it will call your SavedModel through one of its named signatures. In Python, you can test them as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-04T02:32:16.714809Z",
     "iopub.status.busy": "2021-03-04T02:32:16.714009Z",
     "iopub.status.idle": "2021-03-04T02:32:17.020453Z",
     "shell.execute_reply": "2021-03-04T02:32:17.019965Z"
    },
    "id": "0FdVD3973S-O"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: this is such an amazing movie! : score: 0.999552\n",
      "input: The movie was great!           : score: 0.991094\n",
      "input: The movie was meh.             : score: 0.645052\n",
      "input: The movie was okish.           : score: 0.007991\n",
      "input: The movie was terrible...      : score: 0.000922\n",
      "\n"
     ]
    }
   ],
   "source": [
    "serving_results = reloaded_model \\\n",
    "            .signatures['serving_default'](tf.constant(examples))\n",
    "\n",
    "serving_results = tf.sigmoid(serving_results['classifier'])\n",
    "\n",
    "print_my_examples(examples, serving_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continued Learning\n",
    "\n",
    "In this lab, we chose small BERT to train our text classifier. There are other pre-trained BERT models which you can find here. Consider experiementing with some of these. However, remember that the bigger the model you choose to fine-tune, the longer it will take to train.\n",
    "\n",
    "There are \n",
    "  - [BERT-Base](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3), [Uncased](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3) and [seven more models](https://tfhub.dev/google/collections/bert/1) with trained weights released by the original BERT authors.\n",
    "  - [Small BERTs](https://tfhub.dev/google/collections/bert/1) have the same general architecture but fewer and/or smaller Transformer blocks, which lets you explore tradeoffs between speed, size and quality.\n",
    "  - [ALBERT](https://tfhub.dev/google/collections/albert/1): four different sizes of \"A Lite BERT\" that reduces model size (but not computation time) by sharing parameters between layers.\n",
    "  - [BERT Experts](https://tfhub.dev/google/collections/experts/bert/1): eight models that all have the BERT-base architecture but offer a choice between different pre-training domains, to align more closely with the target task.\n",
    "  - [Electra](https://tfhub.dev/google/collections/electra/1) has the same architecture as BERT (in three different sizes), but gets pre-trained as a discriminator in a set-up that resembles a Generative Adversarial Network (GAN).\n",
    "  - BERT with Talking-Heads Attention and Gated GELU [[base](https://tfhub.dev/tensorflow/talkheads_ggelu_bert_en_base/1), [large](https://tfhub.dev/tensorflow/talkheads_ggelu_bert_en_large/1)] has two improvements to the core of the Transformer architecture.\n",
    "\n",
    "The model documentation on TensorFlow Hub has more details and references to the\n",
    "research literature.\n",
    "\n",
    "Aside from the models available above, there are [multiple versions](https://tfhub.dev/google/collections/transformer_encoders_text/1) of the models that are larger and can yeld even better accuracy but they are too big to be fine-tuned on a single GPU. You will be able to do that on the [Solve GLUE tasks using BERT on a TPU colab](https://www.tensorflow.org/tutorials/text/solve_glue_tasks_using_bert_on_tpu)."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "classify_text_with_bert.ipynb",
   "toc_visible": true
  },
  "environment": {
   "name": "tf2-gpu.2-1.m65",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m65"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
